Results showed the National Party of Germany (NPD), which advocates closing German borders to immigrants, won 7.2 per cent of the vote in Mecklenburg-Vorpommern, a northeastern state on the Baltic Sea which borders Poland.
The result would allow the NPD to enter the regional assembly, making Mecklenburg-Vorpommern the third state in the former communist east with far-right representation.
The result alarmed mainstream politicians and Jewish groups, who called on the federal government to renew its bid to ban the party after a previous attempt failed.
Dieter Graumann, vice-president of the Central Council for Jews in Germany, said: "The government must look for ways to impose a ban."
Final results gave the Social Democrats (SPD) 30.2 per cent in Mecklenburg, down from nearly 41 per cent in the state vote in 2002. It was uncertain if they would continue their local coalition with the former communist Left Party or ally with Merkel's Christian Democrats.
Election data showed 15 per cent of 18- to 24-year olds voted for the NPD in Mecklenburg.
In a federal election exactly one year ago, Merkel won a disappointingly narrow victory over Gerhard Schroeder, her predecessor, forcing her into a coalition with his SPD party.
Many Germans had hoped her "grand coalition", with its big majorities in both houses of parliament, would be able to push through crucial reforms.
But it has been plagued by infighting and struggled to deliver promised changes to the healthcare system. Experts say this has helped boost smaller parties such as the NPD.
In Mecklenburg-Vorpommern, the state where Merkel has her local constituency and where she hosted George Bush for a barbecue in July, the economy is weak and nearly one in five is without work, and the jobless rate hovers above 30 per cent in some areas. Like other states in the former communist east, it has seen many leave in search of jobs.
Those conditions have provided fertile ground for the NPD, which was accused of using thugs to bully rival parties during the campaign. Its leaders have played down Nazi responsibility for the second world war and questioned the extent of the Holocaust.
Several hundred protesters gathered outside the parliament building in the Mecklenburg state capital Schwerin, waving "Nazis Out" posters on Sunday.
The federal government has compared the party to the Nazis of the 1920s and tried to ban them in the country's highest court in 2003. It failed after members of the party who testified turned out to be informants planted by the police.