The decision paves the way for an independent inquiry, long resisted by the government, into the death of Baha Musa, an Iraqi hotel receptionist who died in September 2003 after being detained by British troops.
The law lords' ruling, by a majority of four to one, followed an appeal by the ministry of defence. A lower court will now decide if a public inquiry goes ahead.
Phil Shiner, a lawyer representing Musa and other applicants, said the ruling meant the government must order "a full, public and independent inquiry into what went wrong".
"It seems clear from the public record that serious errors of judgment have been made at senior levels both within the military and the government," he added.
Separate cases also involving deaths in Iraq and also before the House of Lords on Wednesday, were rejected.
Shiner, who also brought those cases, said he would now appeal to the European Court of Human Rights in Strasbourg.
"[The decision] ensured that there can never be a British Guantanamo anywhere in the world"
Shami Chakrabarti, rights group Liberty
A separate claim has been filed for Baha in a UK court seeking punitive damages from the government.
In response, the defence ministry said European laws had applied in Musa's case and that it was sufficient that a court martial this year had found one soldier guilty of unlawful conduct.
"The court martial was a thorough and effective investigation into the circumstances of Mr Musa's death and we maintain there has not been a procedural breach," a spokesman said. "The House of Lords did not call for a public inquiry."
Musa, 26 when he died, was detained with several other Iraqis when British troops raided a hotel in the southern city of Basra.
They were kept blindfolded, put in stress positions and beaten during 36 hours of detention, the court martial this year heard. Musa died after suffering 93 injuries, including broken ribs and a broken nose.
Elated human rights campaigners read Wednesday's ruling as holding the government accountable and responsible over any British detention facility anywhere in the world.
Shami Chakrabarti, of rights group Liberty, said the decision "ensured that there can never be a British Guantanamo anywhere in the world".