Election officials said voter turnout stood at 56.7 per cent of the 6.6 million-strong electorate, one hour before the close of voting.
The West said Serbia must vote for reformers if it wants to pursue European Union candidacy.
Brussels froze talks on closer ties last year and says it will restart them only when Ratko Mladic, the former Bosnian Serb general accused of genocide, is found and put on trial.
Michael Polt, the US ambassador to Serbia, said the "elections are giving the people of Serbia a chance to clearly signal that they want a Euro-Atlantic future, and that they want it now".
He said Serb voters had an opportunity "to clearly say that they do not share the retrograde vision of extremists who would be happy to turn Serbia into an isolated island blinded by nationalism within a sea of new member states of the EU and Nato".
The Democratic Party of Serbia, the party of Vojislav Kostunica, Serbia's prime minister, is seen as equally likely to support either of its rivals in forming a government.
The new government faces having to implement more painful economic changes and deal with two weighty international issues: the future of the breakaway Kosovo province and the handover of Mladic.
The UN is expected to rule this year on the fate of Kosovo.
The West favours granting independence to its majority ethnic Albanians as they have demanded since 1999, when Nato bombs drove out Serb forces accused of killing civilians while fighting an insurgency.
The main parties all say that they will not accept the loss of Kosovo, but the Democratic party of Boris Tadic, the Serbian president - the party favoured by the West - has come closest to telling Serbs that it might be inevitable.
Some Western officials have accused hardline nationalists in Serbia's military and police of helping Mladic to hide and thus evade trial by the UN war crimes tribunal in The Hague.
The Democratic party says arresting him is a priority. However, the Radicals, who consider Mladic a hero, are unlikely to deliver him.