Students said the woman had ignored their warnings to close her business.
Their defiance of the government has exposed its failure to regulate Pakistan's thousands of religious schools, even in the federal capital.
Jihad
Abdul Rashid Ghazi, vice-principal at the Jamia Hafsa seminary, threatened jihad unless the women teachers were freed by 4pm (11:00GMT).
He said this was in line with a religious decree issued by the Lal Masjid mosque's prayer leader - Ghazi's brother, Abdul Aziz.
"These vulgarities [brothels] are destroying society and the decree says that in this situation, jihad is the only way," he said, without specifying what that would entail.
"They [police] have arrested our respected, veiled teachers for a corrupt woman."
The deadline passed without any immediate action, but earlier, male seminary students commandeered two police vehicles and beat another plainclothes officer with sticks.
There was no immediate comment from police. However, a police officer confirmed a number of the Jamia Hafsa seminary's teachers had been arrested on Wednesday for holding an alleged brothel owner known as Aunty Shamim.
'Unreasonable'
Authorities are holding negotiations with school administrators to hand over the woman to police but they are "being unreasonable," the officer said.
"They have taken the law into their hands."
Female seminary student Seema Zubair, 20, told The Associated Press the students abducted Shamim, her daughter, daughter-in-law and six-month old granddaughter.
"We are not authorised by the government to arrest them, but we are authorised by our religion, because this [prostitution] is evil."
She said they would not be harmed and would be released if they promised to close the brothel.
The Lal Masjid mosque in Islamabad and associated seminaries have a reputation for preaching an austere interpretation of Islam, and are also known for their links to an outlawed group Sipah-e-Sahaba, accused in sectarian attacks on Shia Muslims.