|Odinga, left, has called off planned protests against the disputed election at the request of Annan [AFP]|
Onlookers clapped and cheered as Kibaki and Odinga shook hands after the discussions.
Odinga said talks would continue until a solution was found.
"My team and I will spare no effort to resolve this crisis," he said.
Kibaki, for his part, vowed to lead the country to unity.
"I appeal to all Kenyans to remain calm and to shun violence as we endeavour to find solutions," he said.
"I am confident that together, our experience, unity and determination will make it possible for us to overcome the challenges."
But soon after the meeting, ODM's secretary-general accused Kibaki of undermining Annan's mediation efforts by using the participants to validate his presidency.
Anyang Nyongo said: "True to his fraudulent character, Mr Mwai Kibaki abused the occasion by attempting to legitimise his usurpation of the presidency."
Plea for calm
Annan called the meeting and the commitment to dialogue "a very encouraging development".
He said: "I want to join my brothers in appealing for calm."
Al Jazeera's Mohammed Adow, reporting from Nairobi, said Annan was "very much respected".
He said: "Annan seems to be succeeding where mediation efforts before him have failed even to get the two men ... to agree to meet.
"The fact that he also has the support of the African Union, the United Nations and the United States as well as the European Union, seems to have really given him the backing he needed to move this process forward."
Earlier on Thursday, Annan met Kibaki amid fresh clashes that have killed at least 12 people, adding to a post-election death toll of nearly 800.
The crisis has displaced a quarter of a million people in addition to the deaths and material damage.
Riot police and protesters on Wednesday fought running battles at a funeral procession held by the opposition for slum residents killed in violence in Nairobi.
Yoweri Museveni, the Ugandan president who was among the first leaders to recognise Kibaki's legitimacy after the contested poll, also met Kibaki on Wednesday.
Odinga, responding to Annan's request during a meeting on Wednesday, called off mass rallies scheduled for Thursday.
Police had banned the protests after clashes at demonstrations last week killed 80 people.
The Kibaki-Odinga meeting raises hopes after a human-rights watchdog accused ODM officials of organising tribal violence in the Kenya's Rift Valley, backing up government allegations.
Separately, at least 12 people were killed in overnight fighting between rival tribes and gangs, Kenyan police said on Thursday.
Six were hacked to death in Kaptembwa and two others in Bahati, two settlements near the town of Nakuru, northwest of Nairobi, a police commander said.
Police shot dead two men in the central town of Limuru when members of Kibaki's Kikuyu tribe started evicting residents from rival tribes from their homes, he said.
Source: Al Jazeera and agencies