Experts say Badawi's continued leadership could be in jeopardy if BN's majority falls back below 80 per cent, or around 178 seats, in the new 222-seat parliament.
A two-thirds majority is needed to change the constitution.
In the last election in 2004, BN won nearly 64 per cent of the vote and more than 90 per cent of the seats in parliament.
The Pan-Malaysian Islamic Party (PAS) scored wins in the northern states of Kedah and Perak and retained power in its stronghold in northeastern Kelantan state.
The Chinese-backed Democratic Action Party (DAP) won Penang, an industrial state that is home to many multinational firms.
DAP and PAS also joined the People's Justice Party, or Parti Keadilan, to take control of the industrial state of Selangor and almost all the seats in capital Kuala Lumpur.
Samy Vellu, works minister and chief of the Malaysian Indian Congress, one of the BN's constituents, lost the seat he had held for nearly 30 years.
Two other cabinet ministers, both ethnic Malays, also lost.
M Manoharan, a detained ethnic Indian activist and lawyer, won a parliamentary seat despite being held under internal security laws for organising a major anti-government protest last year.
Large crowds had flocked to opposition rallies during the campaign, especially ethnic Chinese and Indian voters unhappy with the BN government, dominated by politicians from the Muslim majority of ethnic Malays.
Ethnic Chinese and Indians make up about one-third of the population and many complain of discrimination by the government in favour of ethnic Muslim Malays, in terms of education, jobs, financial assistance and religious policy.
Scattered incidents of violence were reported from different parts of the country on Saturday.
In the northeastern Terengganu state, police fired tear gas to disperse a crowd of 300 opposition supporters.
Twenty-two people were arrested after PAS supporters stopped several buses and cars which they suspected were transporting BN supporters pretending to be voters from the district.
Total seats in parliament: 222
Seats declared (till 2145 GMT): 219
Barisan Nasional (BN): 137
Parti Islam se-Malaysia (PAS): 23
Democratic Action Party (DAP): 28
People's Justice Party
Musa Hassan, a local police chief, said: "The PAS supporters threw stones at police vehicles, forcing the police to release tear gas to control the situation."
In 1969, after BN suffered a major electoral setback, race riots broke out in which hundreds of people were killed and a two-year state of emergency followed.
The elections have also been marred by allegations of vote rigging.
Anwar Ibrahim, a prominent opposition figure whose wife heads Parti Keadilan, condemned the phenomenon.
"We need to teach these cheaters a lesson, by coming in big numbers, enough to deny them, and thwart their plans to rig the vote," he said.
After casting his vote, Nik Aziz Nik Mat, the PAS leader, said his supporters had found a BN member in possession of 28 identity cards.
Earlier, Tricia Yeoh from Malaysia's Centre for Public Policy Studies told Al Jazeera that Saturday's election would be drastically different to the last.
"There has been, in the past four years, an increasing disquiet in almost all sectors of society," she said.
"This is due to a number of reasons ... the failure of the administration to curb corruption [and] the growing income disparity between different societies in Malaysia."
Yeoh said Malaysia's electoral system is in need of reform because of conditions that are unfavourable towards the opposition.
"There is media bias. You have gerrymandering and non-transparent administration procedures," she said.