The more powerful lower house, however, where Abe's ruling coalition has an overwhelming majority, was not up for grabs, and the prime minister said he would stay in office.
'I will continue'
"I will continue to assume my responsibility as prime minister because my task of nation-building has just begun," a sullen-looking Abe said in a television interview on Sunday.
Previous prime ministers have resigned after upper house defeats that were less severe.
Abe assumed power in 2006 on a mission to build a nation "prouder of its past", but he has come under fire over a number of scandals, including the suicide of his agricultural minister and the mismanagement of Japan's pension system.
Analysts said that while the Liberal Democrats were worried that they did not have anyone better than Abe, it would be impossible for him now to govern.
"Abe is finished," said Gerald Curtis, a Japan expert at New York's Columbia University. "I think if he's smart, he'll quit tonight."
Curtis added that "it will just be more and more chaotic politically" the longer he stays.
Taro Aso, Japan's foreign minister, has made it clear that he would like to succeed Abe, but critics say he shares much of the same agenda and is prone to gaffes.
Opposition parties have seized on Abe's woes to try to win over traditional supporters of the Liberal Democrats, such as rural voters who feel left behind by free-market reforms.
Yukio Hatoyama, secretary-general of the Democratic Party of Japan, said: "We realised how strong people's dissatisfaction has been ... People have high expectations for us."
The opposition victory may make it harder for Abe to push through his agenda, although an opposition-led upper house can be overridden by the lower house, where Abe inherited an overwhelming majority from his predecessor, Junichiro Koizumi.
Abe was initially popular when he succeeded Koizumi about 10 months ago, but his approval ratings have since taken a nosedive.