One local police chief said Ghazi had freed two policemen on Sunday after talks with officials. He did not say when the two remaining policemen were expected to be freed.
In recent months, the Lal Masjid mosque has challenged the authority of President Pervez Musharraf's government.
The release of the men came after Musharraf said in an interview aired by Pakistani television on Friday that militancy in Pakistan was increasing, and that there was "a need to strongly counter it."
Cleric defends students
Ghazi defended his students' decision to capture the policement, saying they detained the four officers because they were standing outside the seminary despite an agreement with authorities that police would not be deployed there.
He said the abductions were in retaliation for the detention of eight or nine seminary students by intelligence agents in the past two weeks. He said they freed the two policemen after police agreed to free their students.
"The remaining two policemen will also be freed soon," he said.
The seizure of the police came as critics accused Musharraf of appeasing religious vigilantes, despite concerns that pro-Taliban hard-liners, intent on enforcing a stringent version of Sharia, or Islamic law, have been gaining influence in Pakistan.
In April, female students at Lal Masjid kidnapped an alleged brothel owner and forced her to make a confession as part of an anti-vice campaign.
The mosque later declared it had set up its own Islamic court, and threatened to close many music and movie shops.