More than 90 per cent of the 55,000 registered voters in South Ossetia had voted, the government said, two hours before polls closed on Sunday.
Populated by ethnic Ossetians and Georgians, the former Soviet region has created tension between Moscow and Georgia with Russia backing Ossetians who wish to reunite with it.
The referendum coincides with a presidential vote in the republic which Eduard Kokoity, the incumbent, is expected to win.
Though the polls, like the republic, are not recognised by the international community, South Ossetia's de facto leadership has announced that the referendum is a first step to achieving international acceptance and eventual union with Russia.
International mediators said the poll will heighten tensions in the region, with the Organisation for Security and Cooperation in Europe, OSCE, calling it "counter-productive".
Jaap de Hoop Scheffer, Nato's secretary general, said on Saturday: "I join other international leaders in rejecting the so-called referendum and elections [in Ossetia].
"Such actions serve no purpose other than to exacerbate tensions in the South Caucasus region."
Ossetia has alleged that it had uncovered a Georgian plot to assassinate the separatist leader Kokoity, who told the media that his government had prevented a "coup."
A South Ossetian officer casts
his ballot on Sunday
Polls in Sunday's referendum opened at 0500 GMT and are due to close 12 hours later.
About 55,000 people are registered to vote, the region's elections committee has said.
Up to 100,000 people live in South Ossetia, which broke away from Georgia in a brief war in 1992.
Sunday's referendum comes amid a diplomatic crisis between Tbilisi and Moscow that began last month when Georgia arrested and expelled four Russian officers it accused of spying.
Russia is seeking to double the price of natural gas it exports to its neighbour and has cut off transport links with it, worsening its poor economy.
Georgian officials accuse Russia of using its economic weight and the upcoming referendum to put political pressure on Tbilisi as punishment for the country's pro-Western course, which includes seeking to join the European Union and the US-led Nato military alliance.
Preliminary results will be announced later on Sunday night.