The exchange suspended trading at 2.40pm (0540 GMT), 20 minutes before the normal closing time, after the number of transactions threatened to exceed its computer system's capacity of 4.5 million trades per day.
The Tokyo Stock Exchange (TSE) has been hit by a series of system problems in recent months, including a glitch that halted trading for almost a full day late last year.
Analysts said Wednesday's shutdown would deal another blow to the image of the exchange, which has plans to list its own shares.
Tsuyoshi Nomaguchi, strategist at Daiwa Securities, said: "This is clearly about the TSE's system risk. This is an event unheard of before.
"Investors, who have traded stocks on the assumption that the TSE system does not break down, are running away from the market and the selling pressure will continue for the rest of the day."
Taizo Nishimuro, the exchange president, told a news conference earlier that the bourse would also consider shortening trading hours on Thursday and beyond if necessary. "The recent spike in orders is extraordinary," he said.
The number of transactions had reached about 4 million by
News that the exchange was considering a shutdown accelerated selling across the board, pushing down the Nikkei share average by more than 4% to as low as 15,059.52 before it recovered slightly to finish the shortened session down 2.94% at 15,341.18.
That was the Nikkei's biggest one-day fall since 18 April 2005, when it fell 3.8%. The broader Topix index fell 3.49% to close at 1,574.67.
The three-day sell-off has wiped out more than $300 billion in shareholder value - equal to about the gross domestic product of Sweden.
Yen under pressure
The share price tumble also hit the yen, which fell to a day's low of ¥115.88 to the dollar, before recovering to around ¥115.60.
Ken Masuda, a senior dealer at Shinko Securities, said: "The problem has caused a selling climax. Everyone is throwing in sell orders. The system is jammed and orders aren't making their way through.
"Even after five minutes, orders aren't going through. This is ridiculous."
The investigation into Livedoor
has sparked the panicked selling
Investigators from the Tokyo District Prosecutors' office and the Securities and Exchange Surveillance Commission raided the Tokyo headquarters of Livedoor late on Monday on suspicion that the company had spread false information to investors.
Newspaper reports on Wednesday said the company was also suspected of tampering with its financial reports.
The investigation has scared off individual investors, who
were a major factor in the Nikkei's 40% rise last year. Only last Friday, the Nikkei hit a five-year closing high.
Nomaguchi, of Daiwa Securities, said: "Until the investigation on Livedoor and the fate of the company becomes clear, selling pressure on the overall Tokyo stock market will likely stay. But I think the market will recover after that."