The public broadcaster NHK projected that the LDP-led coalition won anywhere between 31 and 43 seats out of the 76 it was defending.
Private broadcaster TBS put the coalition's win at 34 seats, while Nippon Television gave a figure of 38.
All predictions are well below the 64 seats the LDP would need to maintain a majority.
"The results appear to be very severe," Nobuteru Ishihara, the deputy LDP chief said at his party’s headquarters.
In his first 10 months as prime minister Abe has championed building a more assertive nation proud of its past, dubbed a “beautiful Japan”.
But Paul Allen, Al Jazeera's correspondent in Tokyo, says a beautiful Japan appears at the back of most voters’ minds as Abe has come under fire over a raft of scandals including the government's mismanagement of the pension system.
It was the first time in nine years that the LDP lost control of a house.
Abe's 10-month tenure has been
dogged by scandal [AFP]
The party has not lost a majority in either house since 1998 elections, but Sunday's defeat was on course to be even worse.
Yoshio Yatsu, LDP's committee chief for the elections, said Abe would consult with the party on the next move.
"The ballot counting has just started. We will never know until all the votes are counted," Yatsu said.
"We faced a hard battle in this election because of the scandals over the pension agency and political funds."
The defeat does not automatically mean Abe will leave his post after only 10 months in the job as his Liberal Democrat-led coalition enjoys a large majority in the more powerful lower house inherited from his predecessor Junichiro Koizumi.
However prime ministers have traditionally resigned to take responsibility for defeats in upper house polls in the past.
Despite achieving their biggest success since forming in the 1990s Bryan Walsh of Time magazine says the Democratic Party of Japan, the biggest opposition bloc, would not necessarily be a better alternative to Abe.
Nor, he says, would Taro Aso, the current foreign minister who is Abe’s most likely successor from his own party as he shares the majority of his conservative policies.
Nearly 105 million people were eligible to vote in the election for half the seats in the 242-member upper house of parliament.
"I said no to the Liberal Democratic Party. I said no to Abe," Keiko Yutani, a 60-year-old language teacher, said as she cast her ballot near Tokyo's giant Tsukiji fish market.
"I'm extremely angry at Abe's cabinet," she said. "I can't leave my pension funds to them."
Izuru Makihara, a professor of politics at Tohoku University, said Abe stumbled by trying to address the emotionally charged history issues close to his heart alongside bread-and-butter issues.
The end result was "a failure to convey a clear message to voters," he said.