Gilani ordered the release of all judges.
He also said he will ask parliament to pass a resolution seeking a UN inquiry into the assassination of Bhutto.
"Today, democracy has been restored thanks to the great sacrifice of Benazir Bhutto," Gilani told parliament on Monday.
Bhutto, a former prime minister, led the Pakistan People's Party (PPP) until her killing during the election campaign in December last year.
"I will demand the immediate release of all the arrested judges," Gilani said.
Shortly after, police started removing the barricades outside Chaudhry's home.
Chaudhry appeared in public for the first time since November on the balcony of his house, flanked by his wife and children.
He waved to supporters, many of whom were shouting slogans calling for the resignation of Musharraf.
"I am thankful to the entire nation which has struggled for the last five months for the rule of law," Chaudhry said.
State media quoted a senior Islamabad administrator as saying that "all deposed judges are free to move".
Raja Assad Hameed, a correspondent for The Nation daily, told Al Jazeera that the reinstatement of the fired judges appears to be likely.
"It stands as the most popular demand of the masses, and of the parliament," he said.
|Chaudhry has been under house arrest since|
he was sacked in November last year [AFP]
"Many politicians who were pro-Musharraf and backed the detention of the judges, now have changed sides."
The PPP-led coalition that won the election had vowed to reinstate the sacked judiciary.
If the new government honours that pledge, the judges could still declare Musharraf's re-election illegal and remove him from the presidency.
Chaudhry's suspension in November last year had sparked the country's biggest political crisis since Musharraf seized power in a bloodless coup in 1999.
Musharraf said Chaudhry was sacked over allegations of impropriety and misconduct.
After Monday's developments, one of Chaudhry's aides said that he was grateful to Gilani.
"I have just met the chief justice," Athar Minallah told the AFP news agency. "He has thanked the prime minister and the new assembly and he has also said that he is praying for the democratic institutions."
Gilani on Monday won 264 votes in the 342-seat lower house of parliament, while Chaudhry Pervaiz Elahi, a senior leader of the pro-Musharraf Pakistan Muslim League (PML-Q) party, secured 42 votes in the poll on Monday evening.
Fahmida Mirza, the parliament speaker, said: "Yousaf Raza Gilani commands the majority of the members. Please come forward and take the seat of leader of the house."
Gilani was jailed in 2001 by the Musharraf government for making illegal appointments but was freed in 2006.
He says the charge was politically motivated.
Al Jazeera's Kamal Hyder, reporting from Chaudhry's residence in Islamabad on Monday, said Gilani's appointment as prime minister is seen as a glimmer of hope for Pakistan.
"This [call for judges release] is a victory for the electoral process, and those who had campaigned for the restoration of the judiciary," he said.
"People gathered here are calling for Musharraf's resignation. The question remains, however, is how Musharraf is going to handle the rapidly growing discontent with his policies."
Hyder also said that the newly elected coalition will isolate Musharraf.
"The president asked the public to vote for him in the elections, and an overwhelming majority did not," he said.
"Gilani will bear many responsibilities, and he will have his work cut out, attempting to restore the relevance and the supremacy of the parliament."