Rajoelina had given Ravalomanana four hours to dissolve the government as well as giving up the leadership of this poor Indian Ocean island of 20 million people.
Rajoelina emerged on Saturday after two weeks in hiding to address supporters saying: "There is only one solution - the resignation of Marc Ravalomanana."
The military appeared to offer no resistance as the opposition entered the prime minister's office to name Monja Roindefo Zafitsimivalo as their premier.
|Ravalomanana has admitted to making mistakes but refuses to step down [AFP]
Rajoelina, a former disc jockey who became mayor of the capital Antananarivo, also declared himself president on Saturday - a move he has made once before despite being too young under the country's laws to take the post.
Solo Ramarolahy, a journalist who works for Radio Madagascar, told Al Jazeera the situation in Antananarivo was "confusing".
She said the opposition insisted they were "taking charge" even though the president defied calls to leave.
On Thursday the opposition had proclaimed its own commander-in-chief of the armed forces, again without resistance, and on Friday said it had gained control of military vehicles.
This prompted the president to go on national radio to warn against any attempt on his life.
At Saturday's rally, the head of the national assembly, formerly a Ravalomanana ally, called on the president to resign.
Rajoelina has said he hopes a handover of power would be peaceful.
"I have clean hands," he said. "I have no intention of killing him [Ravalomanana].
"I have no intention of sending in tanks and soldiers."
Protests have occurred across Madagascar since January, with Rajoelina leading citizens against what he calls a dictatorship.
Earlier in the week, Ravalomanana admitted to making mistakes during the political crisis, in which more than 100 people have been killed.
The deaths happened after security forces were ordered to crack down on anti-government protests.
The security forces have increasingly turned against Ravalomanana as pressure grows for him to step down.