Musharraf triggered his worst political crisis on March 9, when he suspended Iftikhar Mohammed Chaudhry, the supreme court chief justice, on unspecified charges of abusing his authority, prompting accusations that he was entrenching military rule.
While lawyers and opposition groups have held dozens of protests across Pakistan, Monday's nationwide rallies were the first called jointly by the Alliance for the Restoration of Democracy, which joins two main opposition parties led by exiled former prime ministers.
About 3,000 people marched through a commercial district of Karachi, Pakistan's largest city, carrying colourful party flags and chanting "Go, Musharraf, go!" and "Friends of America are traitors to the nation."
Hundreds of police in riot gear lined the route but there was no repeat of the trouble that has marked earlier demonstrations.
'He must resign'
"Musharraf has to go now. He must resign and let elections be held under a caretaker government," Raza Rabbani, a senior lawmaker for the Pakistan Peoples Party, told the crowd packed into Karachi's Regal Square.
"Give a final push to a regime that is already on the verge of collapse," he said, drawing louder chanting against the president.
Thousands more people attended similar rallies in Lahore, Multan and other major cities.
It was a significant display by opposition parties who rarely take to the streets en masse.
Curbing democracy
Musharraf has insisted Chaudhry's suspension was not politically motivated and his information minister said on Monday that the agitation was fizzling out.
"With the passage of time, the legal community will understand that there was no ulterior motive to what we have done and that everything that was done was in line with the constitution," Mohammed Ali Durrani said.
"There's no hidden agenda."
Heavy-handed action by authorities, who earlier this month raided a private TV station that aggressively covered the news of the judge's suspension, has added to perceptions that the military leader is curbing the democracy he vowed to restore when he took power in 1999.
Chaudhry's case is before a panel of senior judges, which can either confirm or throw out the allegations he faces. The council's next hearing is set for April 3.