Exit polls say Tomislav Nikolic, a hardline Serb nationalist, has won the most votes but not a majority in the country's presidential election, setting up a run-off with Boris Tadic, the incumbent president.
Opinion polls in the run up to the election gave Nikolic 33 per cent of the vote, against 30 per cent for Tadic.
Sunday's vote, overshadowed by the expected declaration of independence by the breakaway province of Kosovo, registered a high voter turnout.
One hour before polls closed, the turnout stood at 58.2 per cent, the highest since the fall of Slobodan Miosevic, the Yugoslav president, in 2000.
"This vote is the first half of the match," Tadic said after casting his ballot.
"The second half, on February 3, will be a chance to once again decide to conquer Europe, a better life, better salaries and pensions, greater security."
Tadic defeated Nikolic in a run-off in 2004 after his rival topped the poll in the first round.
Divergent views
The incumbent president advocates integration into the European Union while Nikolic believes that Serbia should remain close to Russia, which has backed it over Kosovo, and says he favours a neutral stance between East and West.
Nikolic was an ally of Milosevic and has played on the growing frustration over US and EU backing for Kosovo's independence after the failure of almost two years of negotiations between Belgrade and Pristina over its future status.
"We do not want to argue with the European Union," he said on Sunday. "We need the EU, but not at any price."
Nikolic and his Serbian Radical Party (SRS) have promised tough measures against countries that recognise Kosovo's statehood.