A final tally for Sunday's election is not expected before October 6.
Anton Pelinka, a political analyst from the Central European University, said the prospects for stable government in the near future were "terrible".
"The strength of the far-right parties will make formation of a coalition incredibly difficult if you don't bring either into government," he told Reuters.
Werner Faymann, the SPO's leader, has rejected the possibility of joining forces with either right-wing party.
"I stand by our "no" to a coalition with Freedom or the Alliance. We want a stable government with a broad base ..., not a squabbling government, which is what voters rejected," Faymann said on Sunday.
|The far right Freedom Party took 18 per cent, compared to 11 per cent two years ago [AFP]
The so-called "grand coalition" between the SPO and OVP collapsed in July after months of squabbling.
By retaining their status as the largest single party, the SPO are expected to be asked by Austria's president to form the next government.
The two right-wing parties, which were once united and split acrimoniously in 2005, were not expected to join forces due to past resistance from the FPO.
However, Alliance's Haider said on Sunday that it was something worth thinking about.
Analysts said the right's resurgence would not necessarily mean Austrians were becoming more extremist in their views, but more that voters were likely disenchanted with government gridlock and proposing anti-inflation measures.
"The biggest winner is collectively the radical right ... but that doesn't mean they can come together in a political partnership," said Richard Luther, an expert on Austria at Keele University in Britain.
"I think a grand coalition [of the two biggest parties] is still the most likely, [but] it would be relatively weak in terms of its legitimacy," Luther said.
"This isn't a national catastrophe, it doesn't mean that Austria is a right-wing country. We knew this was coming," Michel Palliardi, 33, who voted for the SPO, said.
"A lot of this was in protest at the [outgoing] government. There is a sense of mistrust."
Voter turnout on Sunday was 71.4 per cent with 6.3 million Austrian entitled to vote, although this did not include about 580,000 postal votes.