Nearly 150 police took part in the operation on Friday to snatch Vasil Mkalavishvili, who had been excommunicated by the Georgian Orthodox Church in 1996 and was hiding with his supporters in a church in the capital Tbilisi.
The police had to use two military trucks to smash through the metal barriers around the building before spraying tear gas and assaulting with truncheons the renegade priest's band of faithful.
Some 20 people were injured in the operation, including children, and 10 were hospitalised.
Dozens of Mkalavishvili's supporters then gathered outside the detention centre of the Georgian interior ministry, where the priest is being held, and said they were launching a hunger strike to demand his release.
"Georgia does not exist right now, it is only another US state..."
The clergyman, who had been sentenced by a Tbilisi court in June to three months in jail for destruction of property, was not present for his trial.
Officially he was on a wanted list but the police did not make any effort to arrest him as he was always accompanied by a few dozen supporters ready to defend him.
On Thursday, Mkalavishvili gave a press conference in Tbilisi in which he violently criticized the new pro-Western government in Georgia led by US-educated lawyer Mikhail Sakaashvili.
"Georgia does not exist right now, it is only another US state, whose governor is George Soros," the US billionaire philanthropist, he said, protesting that the government planned to register the Jehovah's Witnessses.
Father Mkalavishvili had admitted in the past that his faithful had on his orders ransacked the premises of a community of Jehovah's Witnessses and burnt bibles of the Baptist Evangelical Church in Tbilisi.