Tsvangirai, who leads the the Movement for Democratic Change (MDC), arrived at the venue but decided not to proceed with plans to address the crowd.
He announced "the struggle continues" before leaving the venue with his security personnel.
Police armed with batons, guns and tear gas, supported by others in armoured trucks, arrived at the stadium and arrested dozens of supporters arriving for the rally, witnesses said.
Detained by the police
Elias Mudzuri, the MDC's national organising secretary, said about 200 young people who had been on security duty at the sports ground overnight, had been attacked and either driven away or detained by the police.
Police had tried to ban the event, saying they had not been given enough notice about it.
But the Zimbabwe High Court said on Saturday that the government should allow the MDC rally to go ahead regardless.
The party had planned to use the rally, in the capital's Highfield shantytown area, to launch its presidential election campaign.
The election is due in March 2008 but the ruling Zanu-PF party, headed by Robert Mugabe, Zimbabwe's president, plans to put off the elections until 2010.
Political tension is rising in Zimbabwe, where food shortages, inflation at nearly 1,600 per cent and unemployment above 80 per cent have caused great hardship and poverty.
"Zimbabwe is not gaining friends and those friends it had are becoming lukewarm"
Mosiuoa Lekota, South Africa's Defence Minister
Last week, Mugabe was left off a list of African leaders invited by Jacques Chirac, the French president, to attend the France-Africa summit in Cannes.
Four years earlier, the French ignored an EU travel ban on Mugabe, inviting him to attend the conference.
The decision not to invite Mugabe to this year's conference has highlighted the isolation of a leader whose last state visit was to Iran in November.
Eldred Masungure, a professor of political science at the University of Zimbabwe, said: "Zimbabwe is not gaining friends and those friends it had are becoming lukewarm."
Last week Mosiuoa Lekota, South Africa's defence minister, called Zimbabwe "a problem" that needed to be addressed by the whole of the region.