The rallies were held in major cities and towns across the country on Saturday, where speakers denounced Musharraf for breaking his pledge to hang up his army uniform by the end of 2004.
Described as a black day, protesters wore black armbands and waved black flags to symbolise their disapproval.
However, the turnout was marred by severe winter rains continuing in the capital Islamabad and its twin city, Rawalpindi, for the past three days.
Rain hit a rally planned in Peshawar, the capital of North West Frontier Province, where about 300 slogan-chanting people gathered for a demonstration. Heavy rain and snow also kept people indoors in southwestern Baluchistan province.
However, about 400 people attended a rally in the central city of Multan with leaders of the Islamic alliance of Muttahida Majlis-e-Amal (MMA) criticising Musharraf for not abiding by his pledge.
"Musharraf has broken his promise. We will not accept his decision to retain the post of army chief," local MMA leader Hidayat Allah Pasruri told the gathering.
Former prime minister Benazir Bhutto's Pakistan People's Party (PPP) and the multi-party Alliance for Restoration of Democracy (ARD) also joined Saturday's protest after Musharraf's categorical rejection of the opposition demand.
Message to nation
In a nationwide broadcast late on Thursday, Musharraf said he would not hang up his uniform, despite his earlier pledge that he would step down as head of the army.
It was the Islamic alliance itself which in 2003 backed constitutional changes giving Musharraf extra powers - a deal they made in return for his pledge to quit the military by the close of 2004.
However, in November 2004, Pakistan's parliament enacted a controversial law allowing him to backtrack, and Musharraf said in Thursday's address that the constitution allowed him to keep both his military and civilian roles until 2007.
Musharraf, who seized power in a bloodless coup in 1999, accused the opposition, led by the MMA, of destabilising Pakistan by refusing to accept the "ruling of the majority".
Musharraf toppled prime minister Nawaz Sharif in October 1999. He appointed himself president in June 2001 and won a heavily criticised referendum in April 2002, followed by a parliamentary vote of confidence in December 2003.