Sunday's rally comes days after a law was passed allowing him to keep his civilian and military roles.
Thousands of people chanted anti-US and anti-Musharraf slogans at the protest in Multan, witnesses said, following the president's apparent breaking of a promise to hang up his uniform at the end of the year.
The rally, the second in a series of four, was organised by the six-party Muttahida Majlis-e-Amal Islamic alliance and was also attended by leaders of the parties of former prime ministers Benazir Bhutto and Nawaz Sharif.
The campaign against Musharraf began last week with a big demonstration in Pakistan's largest city, Karachi.
Rallies are also planned in eastern Lahore and finally in Rawalpindi, which adjoins Islamabad, on 19 December.
"I have no doubt in my mind that people will force Musharraf to quit as army chief by the end of month," the Islamic bloc's chief, Qazi Hussain Ahmad, told supporters.
"If he does not quit one post, I am confident that he will lose both."
Musharraf overthrew the elected
Pakistani government in 1999
Pakistan's parliament enacted a bill on Tuesday allowing Musharraf to remain army chief, less than 12 months after he said he would become a civilian leader by the end of 2004.
Ironically, his pledge was part of a deal with the Muttahida Majlis-e-Amal in return for its support for constitutional amendments that validated his presidency and gave him sweeping powers, including the ability to dismiss parliament.
Musharraf has since said he needs to keep the army post to continue fighting "terrorism" and seek a settlement with India over the Kashmir dispute.
The new law puts no time limit on how long Musharraf can hold both posts although his term as president is meant to end in 2007.
Musharraf took power in a bloodless coup on 12 October 1999 when he overthrew democratically elected prime minister Nawaz Sharif.
He appointed himself president in June 2001.