Socialist Prime Minister Ferenc Gyurcsany's campaign for a "no" vote in the Sunday referendum, which aimed to embrace Hungarians living abroad after the 1920 carve-up of the Austro-Hungarian empire, appeared set to succeed because of a low turnout.
He insisted that he had won "without joy" and criticised the right wing opposition Fidesz party for playing with the feelings of the Hungarian nation.
"I heard the voters say 'no' to emotion, partiality and to a fruitless looking to the past and to national and social populism," Gyurcsany said after the polls closed.
Final turnout was just over 37%, well below the threshold of 50% to make the referendum binding.
Gyurcsany argued for a "no", saying the country which joined the European Union this year should look to the future. He warned the overstretched budget could be bankrupted by immigrants.
Five million ethnic Hungarians live outside the country and 2.5 million of them live in much poorer neighbours such as Romania, Serbia and Ukraine.
Supporters of a "yes" said it could heal the hurt of a nation which lost two-thirds of its land after the first world war.
Hungary was dramatically reduced
in size after the first world war
"The other side won using a campaign of lies," said Miklos Patrubany, president of the World Federation of Hungarians, who alleged there had been vote rigging.
"This issue enjoys tremendous support and the only reason it could not prevail is due to ... election fraud," he said.
Some 2000-3000 people who had supported a "yes" vote gathered outside Hungary's parliament carrying candles and as the outcome became clear, many wept openly.
With 99.8% of the votes counted, the electorate was split with 48.44% backing Gyurcsany's "no" call and 51.56% backing the right wing Fidesz opposition.
The idea of promoting the interests of Hungarians outside the country has always been closely associated with Fidesz leader Viktor Orban, who lost power in the 2002 elections.