Ali was replaced by Mathew Kirai Iteere, the former head of the General Service Unit (GSU), the police's paramilitary unit, Kibaki's office said in a statement.
No explanation was given for the dismissal.
Ali was criticised for his role in the violence that left around 1,500 people dead following the disputed December 2007 elections.
He came under further pressure in January this year when Philip Alston, the UN special rapporteur on extra-judicial killings, published a critical report on police brutality in Kenya.
Alston cited Ali as one of the main obstacles to his inquiry into the unlawful killings of human-rights campaigners and called for his dismissal.
The UN report on the post-election violence found that killings by police are "systematic" and called for Ali's resignation.
Speaking at a handover ceremony in Nairobi, Iteere said he would tackle organised crime, in reference notably to the Mungiki, a group notorious for mutilating and beheading its victims.
"My main priority will be to tackle the organised crime and criminal gangs that have wreaked havoc in this country and I urge the public to support us in this," he said.
For his part, Ali has always denied the allegations against him and defended his record in improving security in the country's streets.
"The situation as it is today cannot be compared to what it was when I took over. Carjackings, robberies, banditry and organised crime have been successfully brought under control," he said.
The US welcomed Ali's removal from the police, which has consistently been ranked the most corrupt of the country's institutions.
|Ali was criticised for his role in Kenya's post-election violence in 2007 [File: EPA]
"The United States Government has consistently made clear the need for fundamental reforms to strengthen the rule of law in the police force and the judiciary," the US embassy said in a statement.
"The changes announced today in police leadership are a potential first step."
The Mungiki, which emerged in the 1990s, claims to have two million members in central Kenya and in the Rift Valley provinces and says it advocates a return to tribal traditions and customs.
But the authorities have accused the group of being a "mafia" involved in extortion, racketeering, protection rings, kidnapping and murder.
The group was banned in 2002.
Most of its members are from the Kikuyu, Kenya's largest ethnic group, and are said to have links to politicians and powerful families.
Two years ago, police allegedly killed hundreds of its members in a crackdown in Nairobi.