The overwhelming "no" vote on Tuesday sent jubilant anti-Kibaki crowds dancing into the streets and may lead to a realignment of Kenya's political forces before a 2007 election.
Opponents of the new constitution, proposed by Kibaki to replace one dating to independence from Britain in 1963, won 58% of six million votes cast.
Critics say the proposed text failed to curb the president's huge powers, a touchy issue for Kenya's 32 million people fractured by years of "Big Man" rule.
In an address to the nation, Kibaki said: "The vote has shown clearly that the majority are opposed to the proposed constitution. As we had said before, my government will respect the wishes of Kenyans."
The referendum campaign split Kibaki's ruling coalition, deepened tribal tensions and ignited unrest in which eight people died at rallies in a nation traditionally viewed by the West as a lynchpin of stability in turbulent east Africa.
An orange revolution
"We were voting to show our frustrations, we were rebelling against the government's empty promises," Yonah Opiyo, 23, said amid singing "no" supporters in Nairobi.
Kenya's opposition leader Uhuru
Kenyatta casting his ballot
Electoral Commission figures showed the "no" camp with 3.5 million votes, against 2.5 million for Kibaki's "yes" camp, a larger gap than most analysts had anticipated.
"It's a revolution for the orange," said political analyst Kanyiha Karoti, referring to the fruit symbol adopted by the "no" camp championed by the main opposition Kenya African National Union (KANU) party and a party in the ruling coalition.
A rift opened in the government early on when Kibaki abandoned a pledge to appoint ambitious LDP leader Raila Odinga as prime minister, making him roads minister instead, a move that rankled with Odinga supporters for three years.
Kibaki received an overwhelming endorsement from the Central Province heartland of his own Kikuyu tribe but received uneven support in much of the rest of the country of 32 million.
The vote on the complicated charter - which would replace the existing text drawn up on the eve of independence from Britain in 1963 - came after a tumultuous campaign during which eight people died in riots around rallies.