The fall of Mogadishu came after a 10-day offensive by government and allied forces to reclaim much of the territory seized by the Somalia Islamic Courts Council (SICC) since June.
While Gedi on Thursday celebrated a triumphant return to his home village outside Mogadishu for the first time since 2002, he acknowledged that the chaotic country was far from stable.
The flight of the Islamists was a dramatic turn-around in the Horn of Africa nation, which has not had an effective government since the 1991 ouster of a dictator, after they had spread across the south imposing sharia rule and confined the interim government to its base in Baidoa until less than two weeks ago.
Terrified of yet more violence in a city that has become a byword for chaos, some Mogadishu residents took to the streets to cheer government troops, while others hid. Some SICC fighters ditched their uniforms to avoid reprisals.
Abdirahman Dinari, the government spokesman, said the Islamists had fled to the southern port city of Kismayu and that the administration now controlled 95 per cent of Somalia.
But analysts said a government victory was in no way certain and that the conflict could be about to take another turn.
The prime minister says troops have entered Mogadishu after the Council of Islamic Courts abandoned the city.
Gedi, who met with clan leaders to discuss the handover of the city, said, "we are co-ordinating our forces to take control of Mogadishu".
Gunfire could be heard and looting has been reported in the power vacuum that has followed the departure of the Islamic courts fighters on Thursday.
Mohammed Adow, Al Jazeera's Somalia correspondent, said the Islamic courts' decision to pull out was unexpected.
He said: "They had promised the Somali people to defend Mogadishu to the last man, but this hasn't happened.
|"They had promised the Somali people to defend Mogadishu to the last man, but this hasn't happened"|
Mohammed Adow, Al Jazeera's Somalia correspondent
"As they head for the south, the chances look very remote for the Islamic courts to assemble their fighters again in order to be able to wage any war against the Ethiopian and Somali government fighters."
One former Islamic courts fighter said: "We have been defeated. I have removed my uniform. Most of my comrades have also changed into civilian clothes."
"People are cheering as they wave flowers to the troops," said one resident of the Somali capital, adding that scores of military vehicles had passed the Somalia National University.
Mohamed Jama Fuuruh, a member of the Baidoa-based Somali parliament, said: "The government has taken over Mogadishu. We are now in charge ... There will be no problems and everything will be fine."
Meles Zenawi, the Ethiopian prime minister, confirmed that Somali transitional government forces and Ethiopian troops would pursue the Islamic courts leaders.
"We are discussing what to do so that Mogadishu will not descend into chaos. We will not let Mogadishu burn," he said in Addis Ababa.
Speaking for the transitional government spokesman, Abdirahman Dinari said that it had some way to go towards taking over.
"We are taking control of the city and I will confirm when we have established complete control," he said.
"Our forces effectively control Mogadishu because we have taken over the two control points on the main roads outside the city."
Later Dinari told Al Jazeera the government had declared a state of emergency "to control security and stability".
The Islamic courts' chairman has said that his side's hasty withdrawal was a tactical move in a war that began last week against Ethiopian troops defending Somalia's weak government.
|Local residents worry that former commanders|
may reoccupy their turf in Mogadishu [Reuters]
One resident said: "Uncertainty hangs in the air."
"My worst fear is the capital will succumb to its old anarchy. The government should come in now and take over - this is the best chance they have before the city falls into the hands of the warlords again."
Al Jazeera's Mohammed Adow said that local commanders have already begun taking over parts of the city.
"It looks now that the government has on its hands a very difficult task in pacifying Mogadishu ... It looks like Ethiopian troops will be here for some time to come".
Dinari said Yusuf Abdullahi, the Somali president, and Ali Mohamed Gedi, the prime minister, remained in the transitional government's south-central base in Baidoa and would move to Mogadishu at the earliest opportunity.
The government has long viewed Mogadishu as too dangerous to move to, but its return would be a key step in achieving greater legitimacy.