The opposition also won control of five of the country's 13 states, putting Abdullah's future as leader in doubt.
Barisan has in effect ruled Malaysia since independence from Britain in 1957.
Without a two-thirds parliamentary majority, Barisan can no longer change the constitution or make some key appointments without debate.
In a recorded speech, Abdullah thanked voters for support that enabled his coalition to form the government, and promised to implement its manifesto.
"I would like to emphasise that Barisan and other component parties accept the people's wishes in the election that was conducted smoothly, based on a democratic process that is fair and transparent," he said.
"God willing there will be no disturbance; there will be no chaos."
Malaysia is largely a mix of ethnic Malays, who make up slightly more than half of the population. Ethnic Chinese and Indians together account for about a third.
A protest vote from Chinese and Indians, upset over what they saw as racial inequality in terms of business, job and education opportunities, had been expected.
But Malays, who traditionally support the ruling coalition led by the United Malays National Organisation, voted for the opposition in droves.
Abdullah, who just four years ago led the coalition to a record election victory, now faces a difficult period, although several party leaders are to meet him to pledge their support, a source close to the party leadership said.
His humbling performance at the federal level was compounded by the loss of his own home state, Penang, although he won his own seat there.
Joining calls for Abdullah's resignation was Mahathir Mohamad, his predecessor.
"He should accept responsibility," said Mahathir, who has expressed regret over picking Abdullah as his successor, saying Najib Razak, the current deputy premier, should have taken over.
Malaysia's share market is expected to slide on Monday, as political uncertainty takes hold after the massive protest vote.
The last time the coalition suffered a heavy setback, in 1969, race riots erupted.
"This is probably not good news for the equity market," Tim Condon, Singapore-based head of Asia research for investment bank ING told Reuters.
Pankaj Kumar, chief investment officer at Kurnia Insurance, said "the political stability has finally been challenged".
A sales broker said he expected the benchmark Kuala Lumpur Composite Index to fall 50 to 100 points, or about 8 per cent, on Monday.
The Democratic Action party (DAP) won Penang while the Islamist party, PAS, scored victories in the northern states of Kedah and Perak and strengthened its grip on Kelantan.
DAP and PAS also joined Parti Keadilan to wrest control of the industrial state of Selangor and almost all the seats in Kuala Lumpur, the capital, from the ruling coalition.
Now the disparate opposition parties must work together.
Anwar Ibrahim, the former deputy premier turned opposition figure and de facto leader of Keadilan, said he was "playing the role of co-ordinating the opposition outside parliament".
"I am optimistic that we can have a more coherent force," he said.
Keadilan emerged as the biggest opposition party in parliament, with 31 seats.