Political analysts said Saturday's result indicated voters were wary of Chen's plans to create a separate Taiwanese identity.
Beijing was likely to be pleased that the opposition Nationalist Party - which favours closer ties with China - kept control of parliament.
"We don't want war. We don't want our government to take the road of provocation and create tension," triumphant Nationalist leader Lien Chan told a news conference as hundreds of his supporters waved party flags and set off firecrackers outside.
"We hope that we can maintain the status quo between the two sides," he said, referring to relations with Beijing.
There was no immediate reaction from China, which views Taiwan as part of its territory and has threatened to attack if the self-governing democratic island declares formal statehood.
The Nationalists, also known as the Kuomintang, won 114 of 225 legislative seats, including those awarded to its allies, the People First Party and the New Party.
Another two seats were given to Nationalist Party members who had ran as independents.
Chen has been blamed for raising
tensions with China
They defeated Chen's Democratic Progressive Party (DPP) and its pro-independence ally, the Taiwan Solidarity Union, which together won 101 seats.
The rest went to independent candidates and other groups, said the Central Election Commission.
Most analysts had expected neither side to score a majority.
"The people are saying perhaps that Chen's recent statements and his policies are too radical," said Andrew Yang, a political analyst from the Chinese Council of Advanced Policy Studies, a prominent private thinktank in Taipei.
The Nationalist alliance, known as the opposition under Taiwan's complex political system where policies are set by the president who appoints the cabinet, holds 51% of seats in the current legislature.
It has in effect used that advantage to block many of the Chen administration's initiatives.
For example, it blocked a proposed $18 billion special budget to buy advanced weapons from the US, saying they were overpriced.
Chen had hoped to win control of parliament to push through policies analysts saw as likely to antagonise Beijing. He had promised Taiwan's 23 million people a referendum on a new constitution in 2006 if his party won on Saturday.
The Nationalist victory is a setback, as two-thirds of lawmakers need to approve constitutional amendments under a bill passed by parliament in August.
But Chen said he fully accepted the result.
"The end of the election should be the beginning of reconciliation and cooperation," he said. "Let us unite Taiwan, stabilise ties across the Taiwan Strait and work together for economic prosperity."
While some in Taiwan held out hope the election results would spur China to resume fence-mending talks with Taipei - suspended since 1999 - others were less optimistic.
"Even if Chen extends an olive branch to the other side in the foreseeable future, I don't think China will accept it," analyst George Tsai of National Chengchi University said in Taipei.
The elections have deepened
divisions in Taiwan
Beijing is convinced Chen plans to declare statehood before his second term ends in 2008, and China has pointed more than 600 missiles at Taiwan in preparation for possible war.
Security analysts view the Taiwan Strait as a dangerous Asian flashpoint.
Chen has promised not to hold a referendum on independence but is assertive about the island's sovereignty.
Taiwan and China split after the Nationalists lost a civil war in the mainland to the communists in 1949 and fled to the island, bringing with them the Republic of China government.
The legislative campaign, coming hard on the heels of a hotly contested presidential vote, has deepened bitter divisions on the island between those seeking formal statehood and those who do not want to rule out eventual unification with China.
Political violence is rare in Taiwan, but the sharp public divide has exacerbated latent social and political tensions.
Police found packages of suspected explosives at Taipei's main railway station earlier this week, triggering a bomb scare.
Voter turnout was relatively low at 59%, compared to 66% in the 2001 legislative poll.