Kibaki made the announcement on state-owned television on Wednesday.
"Following the results of the Referendum, it has become necessary for me, as the President of the Republic, to reorganise my Government to make it more cohesive and better able to serve the people of Kenya," Kibaki said.
"I have directed that the Offices of all Ministers and all Assistant Ministers become vacant. Consequently, the occupants of the said offices cease to hold their respective offices with immediate effect," he said.
Isiah Kabira, head of the Presidential Press Service, said that Vice President Moody Awori and the top civil servants called permanent secretaries in Kenya, will run the government in the mean time, together with Kibaki.
The referendum on Tuesday dealt a blow to Kibaki's government two years before the next presidential race.
About 57% of voters rejected the referendum, which critics had said gave Kenyans a chance to express their opinion about Kibaki's rule, while 43% supported the draft charter, according to Electoral Commission Chairman Samuel Kivuitu.