The hardline Radical party attracted the most support in the election, achieving 28 per cent of the vote.
But it is unclear whether it can form a government and the party's deputy leader, Tomislav Nikolic, says he does not expect to be invited to form the next government.
Nikolic said he expected the Serbian president, Boris Tadic, "will not give us the mandate".
"Tadic already said last night that he would not respect the democratic principle and he would not give a mandate to the Serbian Radical Party," Nikolic said.
He said the pro-Western parties would have a hard time forming a coalition and forecast that it would be shortlived and that more elections would be held before the end of the year.
Solana denied that the vote demonstrated the failure of Western efforts to quell nationalist feelings in Serbia.
The West told Serbs before the election that they should turn their backs on nationalism if they hoped to join the EU and Nato.
But many Serbs appeared to have ignored those appeals.
Tadic, the West's great hope, led his opposition Democratic party to about 23 per cent of the vote, well behind the Radical party.
The EU froze a stabilisation and association agreement (SAA) with Belgrade in May over its failure to surrender Ratko Mladic, the former Bosnian Serb army chief, who was believed to be hiding in Serbia.
He is wanted by the International Criminal Tribunal for the Former Yugoslavia for war crimes in the conflict in Bosnia in the mid-1990s.
Vojislav Seselj, the Radical party's official leader, remains on trial at the tribunal in the Hague.