Lawyers have led opposition to Musharraf since he fired judges, including chief justice Iftikhar Muhammad Chaudhry, under emergency rule in November.
They have vowed to stage a sit-in outside parliament until the government restores him.
"I can see in you the dawn of a new sun," Sharif told the crowd, which answered with cries of "Hang Musharraf!"
"Your passion will restore the supremacy of the law, restore the judges and punish this dictator."
"We won't let anyone sit in peace until the judges are restored"
Nawaz Sharif, Paksitani former prime minister
Friday's rally, which was held amid tight security, came after hundreds of vehicles completed a 24-hour journey from the eastern city of Lahore on the final leg of a nationwide protest.
Police said around 30,000 people had gathered, including protesters outside the parliament building in the capital.
Protest organisers said the figure was 50,000.
Around 6,000 paramilitary troops and police were deployed in the capital Islamabad in advance of the arrival of the lawyers.
The authorities used barbed wire and shipping containers to block the parliament building and stationed armoured personnel vehicles at several points.
The protest was the largest since Musharraf's allies lost elections in February and were replaced in government by a coalition led by the party of assassinated ex-premier Benazir Bhutto and Sharif's party.
Sharif joined the convoy of vehicles on Friday after addressing the protesters at a rally the previous day.
|The protesters have vowed to continue|
until the judges are restored [AFP]
"We won't let anyone sit in peace until the judges are restored," he said.
Sharif's remarks were seen as a veiled criticism of Asif Ali Zardari, leader of the ruling Pakistan People's Party (PPP) and widower of Bhutto.
"It is so sad and regrettable that even though we have democracy we are still demanding the restoration of the judges," he said.
Al Jazeera's Imran Khan, reporting from Islambad, said the event had been dubbed "extraordinary" by the lawyers' movement.
"This [protest] will go on into the early hours of the morning. Official estimates say there are 10,000 people," said.
"It is a massive threat to president Musharraf's government. He has been consipicous by his absence.
"We have heard nothing from him; he has not appeared on state television or on any local private media channels.
"He is actually holed up in the army house - his private residence in Rawalpindi. He must be very nervous at the numbers of the people that have turned up."
Khan said that the protesters were not just trying to force the restoration of the judges, but they were also putting pressure on Musharraf to step down.
The president has been isolated since his allies were trounced in a February election.
Threat to coalition
But Sharif's participation in the protest also threatens the tenuous unity of the two-month-old coalition government led by the PPP.
Sharif's Pakistan Muslim League Party is in coalition with the PPP but the two leaders have differed on how to bring back the dozens of dismissed judges.
Analysts have suggested that, if restored, Chaudhry may review an amnesty that dropped corruption cases against Bhutto, Zardari and other PPP members.
That, critics say, is why Zardari has been reluctant to restore Chaudhry and his colleagues.