"From now, I'll be giving orders," Rajoelina told the crowd.
"Since the president and the government have not taken their responsibilities, I therefore proclaim that I will run all national affairs as of today," he said.
"A request for the immediate resignation of the president will soon be filed with parliament in order to comply with the legal procedure."
Rajoelina, a former DJ and businessman, has earned himself the nickname "TGV", in reference to France's high speed train.
The mayor has made a series of claims in the last week, saying he is ready to take over an interim government.
But the constitution requires a president be at least 40. Rajoelina is 34.
Saturday's rally ended peacefully, although some protesters threw stones at police in the morning.
Police withdrew from the area and protesters set up a barricade around the square.
Rajoelina's announcement came after a week of civil unrest in the capital.
Violence flared on Monday when protesters set the government broadcasting complex ablaze, along with an oil depot, shopping mall and a private TV station linked to Ravalomanana.
The protests were sparked by the government's decision to close a radio station owned by Rajoelina.
By Friday, the violence had subsided but a tense atmosphere remained on the streets of the capital.
The United Nations, US and Canada have all issued calls for calm in the former French colony in Africa.