Monday's defeat for the proposal strengthens Romano Prodi, the prime minister, who campaigned against the reforms saying they would wreck national unity, weaken the president and cost billions of euros.
For his rival, former prime minister Silvio Berlusconi, the failure of reforms supported by his centre-right bloc could call into question his leadership of the opposition.
Results from all polling stations, except those overseas, showed 61.7% of the voters turned down the plan in a two-day national referendum.
Partial results showed heavy turnout across the country and a "no" vote even in the north, where the centre right's devolution lobby was strongest.
Opponents - including Prodi and many constitutional experts - say change is necessary, but criticise the reforms drawn up by the previous conservative government as being a slapdash, confusing effort that would give too much power to the executive.
The reforms would have strengthened the prime minister's powers, transfer some authority away from Rome to the regions and are meant to speed up the passage of legislation.
The referendum asked for a vote for or against, but the changes would have altered more than 50 of the 139 constitutional articles, representing the biggest change ever made to the document, enacted in 1948.
Francesco Brasi, a 34-year-old bank employee who cast his ballot at a high school in Rome's historic centre, said: "I voted for 'no' because it seems too simple for a reform that is so complicated.
"I listened to all the experts, who were all contradicting each other, and I voted 'no' because this reform is too complicated for a block vote."
The proposed reforms were an attempt modernise an antiquated constitution that limits the prime minister's powers more than other Western governments, reflecting the country's fear of dictatorship after Benito Mussolini's Fascist wartime government.
Prodi will now push forward with his own plans for constitutional and electoral reforms. "Constitutional reforms need the broadest possible support and not just that of the governing majority," he said.
The changes would have given Italy's 20 regions autonomy over health, schooling and policing, a move critics said would mean better services for richer northern regions to the detriment of the poorer south.
Italy has had 61 governments since the second world war.