According to a preliminary forecast from the Centre for Free Elections and Democracy on Monday, Nikolic received around 30.7% of the vote in the election's first round while Tadic received 27.4%.
A Radical Party candidate, Nikolic won the most votes despite the fact that his party is led by war-crimes suspect Vojislav Seselj - who is detained in The Hague.
A Nikolic win in the second round would be a major blow to conservative Prime Minister Vojislav Kostunica, whose ally Dragan Marsicanin came fourth behind business tycoon Bogoljub Karic.
As people elsewhere in former communist eastern Europe voted for the first time in European Parliament elections, Serbia's choice could influence its chances of catching up with its former communist neighbours and joining the wealthy economic bloc.
Diplomats have warned that a Nikolic victory in the presidential race would scare off badly needed foreign investors even though he has toned down his rhetoric and would command only limited powers as president.
The Radicals oppose handing over suspects to The Hague war-crimes tribunal. But compliance with the UN court is a key condition for closer ties with the EU.
Reformer Boris Tadic may pick up
votes from defeated candidates
Leading pro-market reformer, Deputy Prime Minister Miroljub Labus, said he would pull his G-17 party out of government if Nikolic won.
Serbia and Montenegro Foreign Minister Vuk Draskovic said he expected Serbia "to vote for the path towards Europe and not towards the past and isolation".
Surveys published in the run-up to the election predicted Nikolic would face Tadic in the second round, but Tadic could still pick up more votes from supporters of defeated candidates.