Dozens of friends and relatives waited at Sarajevo airport for their arrival, but the men were taken away in an armoured vehicle.
Bosnia's state investigation and protection agency said it would question them, check their identity and release them.
Boudella's three children were visibly upset as the vehicle drove off with their father, while his wife, Nadja Dizdarevic, assured them he would be home soon.
Abdul Aziz, Boudella's son, said: "I still cannot believe this is happening."
Nur Boudella, Boudella's daughter who was only weeks old when he was arrested, said: "I will tell dad that I have missed him."
It was unclear when Lakhdar Boumediene and Saber Lahmar, the other two men ordered released by the US court, might be released.
Boumediene was stripped of his Bosnian citizenship in 2006 and Lahmar never held Bosnian citizenship, although he is married to a Bosnian woman and has two Bosnian children, Oleskey said.
They were among six men picked up by Bosnian authorities in October 2001 and sent three months later to Guantanamo, where they were held for nearly seven years as "unlawful enemy combatants" without being charged with a crime.
Bush said in 2002 that they had been planning a bomb attack on the US embassy in Sarajevo.
But justice department lawyers dropped those accusations when the case went to court, arguing instead that the men had planned to go to Afghanistan to fight against US forces.
Richard Leon, a US district judge, said that charge was based on only one unnamed source whose credibility could not be ascertained.
He ordered five of the men released "forthwith" but ruled that there was enough evidence to continue holding the sixth man, Belkacem Bensayah.
Oleskey said the case against Bensayah was "quite thin as well" and that he hoped the government would promptly comply with the order to release the other two.