Luke Tamborinyoka, an MDC official, said: "Mr Tsvangirai ... was arrested as he was driving out of Highfield. We don't know where he is being held."
Officials had earlier said that Arthur Mutambara, who leads another faction of the MDC, and Lovemore Madhuku, of the pressure group National Constitutional Assembly, were also detained.
Riot police moved in force early on Sunday to head off the prayer rally, which police had said would violate a ban on political protests imposed after opposition supporters clashed with police in Highfield last month.
The organiser, opposition group Save Zimbabwe Coalition, had argued that the ban should not apply to a prayer vigil.
Tear gas
Shop owners in the area closed their stores while hundreds of people wandered the streets under the watch of police units.
Witnesses said later in the day police had fired tear gas at youths who were throwing stones at their patrols, taunting them and defying orders not move around in large numbers.
"There have been several skirmishes between the police and some youths, people throwing stones, and the police firing tear gas," a Zimbabwean journalist who lives in the township said.
Riot police mounted road blocks on major highways into the township and searched vehicles for arms and questioned motorists about where they were going.
Police on Saturday accused some elements in the MDC of hiring and arming "thugs" to attack officers.
"As far as we are concerned that is a political rally ... and we are going to stop that meeting," Wayne Bvudzijena, a police spokesman told a news conference.
Economic crisis
Bvudzijena said he had no immediate information on the arrests.
Officials have increased curbs on opposition movements after violence broke out last month when riot police broke up an MDC rally despite a court order directing that it should be allowed.
State media said officials feared the rally was intended to launch street protests against Mugabe's government.
Zimbabwe has seen political tensions build as it sinks deeper into its worst economic crisis in decades, with inflation above 1,700 per cent, unemployment close to 80 percent and regular shortages of food and fuel.
Mugabe, 83, and in power since independence in 1980, dismisses the MDC as a puppet of Zimbabwe's former colonial master Britain.