Only one of the 26 cantons supported the SVP motion.
"The right to naturalisation has been cemented with today's decision," the SVP said in a statement after its proposal was defeated.
"The negative byproducts of the mass naturalisation of poorly integrated foreigners - violent crimes and social abuse - will rise," the party said.
In the build-up to the referendum, the SVP plastered the country with posters of yellow and black hands grabbing at Swiss passports.
The SVP also sent out literature detailing various crimes committed by immigrants who had been granted Swiss citizenship.
The Social Democratic (SP) party, one of the partners in Switzerland's ruling coalition, said the SVP's "campaign of fear" had failed.
"The rejection of the naturalisation initiative represents the definitive end of the lottery for decisions on naturalisation," the SP said in a statement.
The government had called on voters to reject the proposal, preferring decisions on citizenship to be made by elected bodies with a right of appeal.
The SVP is backed by Christoph Blocher, a billionaire Swiss industrialist, and has increased its power over the last 10 years by focusing its policies on immigration concerns.
More than a fifth of Switzerland's 7.5 million residents are foreigners, according to 2006 data from the federal statistics office.
Switzerland has more foreign residents than almost any other European country, said to be due partly to the difficulty of becoming a naturalised citizen.
Candidates for Swiss citizenship must have lived in Switzerland for 12 years and pass tests on Swiss culture and language.
Last October, the SVP drew accusations of racism by rights groups and the UN during for its election campaign posters that showed a black sheep being kicked off a Swiss flag by three white sheep.