"I congratulate Mr Gedi for his decision," Yusuf told the parliament, and said he would hold wide consultations before choosing a new prime minister.
Gedi has led the embattled transitional federal government since November 2004, but his relations with Yusuf have been tense, with both men coming from Somalia's two main, rival clans.
The prime minister comes from the Hawiye clan, which is dominant in Mogadishu and is the largest in the country.
Yusuf is from what has now become the breakaway northern state of Puntland and comes from the rival Darod clan, the country's second largest.
Al Jazeera's Andrew Simmons, speaking from Nairobi, said: "This doesn't come as a surprise, the prime minister has had differences for a long time with President Yusuf."
"Getting rid of the prime minister, though, doesn't necessarily solve things because the Hawiye clan has to assume the role of prime minister.
"The president can't afford to risk disunity so he will have to appoint someone very quickly from the Hawiye clan."
Relations between the two men worsened earlier this year when they backed different parties interested in exploiting Somalia's oil reserves.
Yusuf had also been pushing parliament to oust Gedi for failing to bring an end to violence in Mogadishu, draft a new constitution and bolster the federal government.
Gedi's critics blame him for being behind the decision to invite Ethiopian forces into Somalia to help the struggling interim government in its battle against the Islamic Courts Union.
But since the fighters of the Islamic Courts, which had controlled large parts of the country, were defeated earlier this year, remnants of the movement and its tribal allies have waged a guerrilla conflict in Mogadishu.
Somalia, a nation of 10 million, has lacked an effective government since the 1991 ouster of Mohamed Siad Barre, the country's then ruler.