He also said he will ask parliament to pass a resolution seeking a UN inquiry into the assassination in December of Benazir Bhutto, who led the Pakistan People's Party (PPP).
Gilani was sworn in as prime minister hours after two senior US diplomats arrived in Pakistan for security talks.
John Negroponte, the US deputy secretary of state, and Richard Boucher, US assistant secretary of state, met Musharraf, Gilani, and Nawaz Sharif, a former prime minister and head of the PML-N party.
While the envoys made no public comment, Sharif said he told them that it was "no longer a one-man show in Pakistan" and that the new parliament would decide how Pakistan should handle its battle with pro-Taliban fighters in the country's tribal belt.
Sharif criticised Musharraf's US-backed policies, saying they had resulted in a wave of suicide bombings that killed Bhutto and many others, and argued that the security of Pakistan must not be sacrificed in order to protect other countries.
He said: "It is unacceptable that while giving peace to the world we make our own country a killing field."
Gilani told parliament on Monday that democracy had been restored to Pakistan.
"I will demand the immediate release of all the arrested judges," Gilani said.
Shortly after, police started removing the barricades outside Chaudhry's home.
Chaudhry appeared in public for the first time since November on the balcony of his house, flanked by his wife and children.
He waved to supporters, many of whom were shouting slogans calling for the resignation of Musharraf.
"I am thankful to the entire nation which has struggled for the last five months for the rule of law," Chaudhry said.
State media quoted a senior Islamabad administrator as saying that "all deposed judges are free to move".
Khalil-ur-Rehman Ramday, one of the judges Gilani released on Monday, told Al Jazeera that the prime minister's decision was "a breath of fresh air".
He said: "The resolve Gilani showed in parliament is certainly a step in the right direction."
Raja Assad Hameed, a correspondent for The Nation daily, told Al Jazeera that the reinstatement of the fired judges appears to be likely.
"It stands as the most popular demand of the masses, and of the parliament," he said.
|Chaudhry has been under house arrest since|
he was sacked in November last year [AFP]
"Many politicians who were pro-Musharraf and backed the detention of the judges, now have changed sides."
The PPP-led coalition that won the election had vowed to reinstate the sacked judiciary.
If the new government honours that pledge, the judges could still declare Musharraf's re-election illegal and remove him from the presidency.
Chaudhry's suspension in November last year had sparked the country's biggest political crisis since Musharraf seized power in a bloodless coup in 1999.
Musharraf said Chaudhry was sacked over allegations of impropriety and misconduct.
After Monday's developments, one of Chaudhry's aides said that he was grateful to Gilani.
"I have just met the chief justice," Athar Minallah told the AFP news agency. "He has thanked the prime minister and the new assembly and he has also said that he is praying for the democratic institutions."
Gilani on Monday won 264 votes in the 342-seat lower house of parliament, while Chaudhry Pervaiz Elahi, a senior leader of the pro-Musharraf Pakistan Muslim League (PML-Q) party, secured 42 votes in the poll on Monday evening.
Fahmida Mirza, the parliament speaker, said: "Yousaf Raza Gilani commands the majority of the members. Please come forward and take the seat of leader of the house."
Gilani was jailed in 2001 by the Musharraf government for making illegal appointments but was freed in 2006.
He says the charge was politically motivated.
Al Jazeera's Kamal Hyder, reporting from Chaudhry's residence in Islamabad on Monday, said Gilani's appointment as prime minister is seen as a glimmer of hope for Pakistan.
"This [call for judges release] is a victory for the electoral process, and those who had campaigned for the restoration of the judiciary," he said.
"People gathered here are calling for Musharraf's resignation. The question remains, however, is how Musharraf is going to handle the rapidly growing discontent with his policies."
Hyder also said that the newly elected coalition will isolate Musharraf.
"The president asked the public to vote for him in the elections, and an overwhelming majority did not," he said.
"Gilani will bear many responsibilities, and he will have his work cut out, attempting to restore the relevance and the supremacy of the parliament."