Bishop Paul Verryn, who has been leading the church's effort to look after the refugees and asylum seekers, accused the police of using excessive force.
He said: "I saw people assaulted when they were put in the vans."
"When I said, 'You can't do this', they told me not to interfere. They pulled me down the stairs by the scruff of my neck and one police officer kicked something at me."
Verryn said he had been told police were looking for illegal immigrants, drugs and weapons.
He also said that the police damaged doors and windows in the church as they searched it.
Outside the building, where about 500 Zimbabweans sleep, hundreds of men were lined up before being put into vans by police.
Verryn said pregnant women were among those targeted in the police raid.
Police officials have so far made no comment.
"This is a church. We thought we were safe," said Fredrick Chibungu, who has been in the country for seven months and is waiting for his asylum papers to be finalised.
"They are going to deport us. There is nothing we can do. It is better to go back home and make a plan to come back," he said.
After an hour police had allowed around 100 women back inside the church because they were looking after a number of children.
Zimbabwe is struggling with an economic meltdown, marked by soaring poverty and inflation, unemployment of around 80 per cent and chronic shortages of food, fuel, water, electricity.
An estimated 4,000 Zimbabweans cross into neighbouring South Africa each day.
The arrivals, however, have stirred resentment among some South Africans, who blame them for contributing to the country's high levels of crime and stealing jobs from locals.