He said religious leaders in the Lal Masjid, or Red Mosque, and adjoining Jamia Hafsa madrasa were hurting the image of Islam.
"Islam is a religion of peace and it does not need Kalashnikovs and sticks," he told the rally.
Mazhar Hussain, a Karachi police official, said about 100,000 people had attended the rally, with many protesters chanting, "No to extremism".
Lal Masjid announced earlier this month that a court had been set up to deliver justice according to Sharia Islamic law. Clerics at the mosque threaten to retaliate with suicide bombers if the government tried to suppress the movement.
Maulana Abdul Aziz, the mosque's chief cleric is a critic of Pakistan's support of the so-called war on terror. Thousands of his male students are often at the forefront of anti-government and anti-US rallies in Islamabad.
Students have also pressured owners of music and video shops to close down their businesses.
In January, girls from the madrasa occupied a nearby public library to protest against the demolition of mosques built illegally on city land and later raided a house they said was a brothel.
Musharraf's government is trying to negotiate a peaceful settlement with the movement in Lal Masjid.