The organisers have held last-minute negotiations with police for permission to hold the meeting, but are unlikely to be successful according to a police spokesman.
Shop owners in the area shuttered their stores and some employed private security guards, while hundreds of people wandered the streets under the gaze of police units.
Police spokesman Wayne Bvudzijena accused the main opposition Movement for Democratic Change (MDC) of hiring and arming "thugs" to attack police.
He claimed police had uncovered a plot by a group allegedly distributing weapons to use against the police, and warned that security forces would be "fully deployed" to prevent the prayer rally.
State radio said the group, called the Democratic Resistance Committee, is fuelling violence in some of Harare's poorer districts and was prepared to use dangerous weapons against the police.
"As far as we are concerned that is a political rally ... and we are going to stop that meeting," Bvudzijena told a news conference.
A spokesman for the Save Zimbabwe Campaign (SZC), which is organising the rally, said it was still going to take place despite the police ban.
They advertised Sunday's event as a peaceful prayer rally open to all Zimbabweans concerned about mounting economic problems and poverty.
"It is still expected to go ahead, but if not we'll have to go back to the drawing board," said Lucky Moyo, a spokesman for SZC. 'Dirty tricks'
|"As far as we are concerned that is a political rally ... and we are going to stop that meeting."|
Zimbabwean police spokesman
Officials imposed a three-month ban on political protests and rallies after last month's violence, which saw riot police use water cannon and tear gas to break up an MDC rally.
State-run media said the rally was intended to launch street protests against Mugabe's government.
The MDC says it has been a victim of a "dirty tricks" campaign by the government in which its officials are accused of violent crimes committed by ruling party youth brigades.
Mugabe, who is 83 and in power since independence in 1980, dismisses the MDC as a puppet of Zimbabwe's former colonial master Britain which opposes him for seizing white-owned commercial farms to give to blacks.
The country is in the grip of its worst economic crisis in decades, with inflation now above 1,700 percent, unemployment of close to 80 percent and regular shortages of food, fuel and foreign exchange.
On Friday, Moragn Tsvangirai, the leader of the MDC, blamed Zimbabwe's neighbours, the Southern Africa Development Community, for contributing to his country's crisis.
He told an annual dinner for foreign correspondents in Johannesburg: "They are tacitly supporting Mugabe on issues where they know Mugabe is not right," alluding to South African president Thabo Mbeki's stated policy of 'quiet diplomacy' with Harare.
"It is time they came on board and spoke with one voice on issues in Zimbabwe," he said.