The Centre for Constitutional Rights (CCR) in Washington, which represents many detainees at the US naval base in Cuba, filed the suit on Wednesday along with a Washington-based law firm on behalf of Shafiq Rasul, Asif Iqbal, Rhuhel Ahmad and Jamal al-Harith, the lawyers said.
Each is seeking $10 million in damages in the action in a Washington DC district court.
They want "damages for torture and other human-rights violations stemming from their arbitrary detention on the island prison camp until March 2003", the CCR said.
In addition to Rumsfeld, the lawsuit names Richard Myers, Chairman of the US Joint Chiefs of Staff; former commander of the detention camp General Geoffrey Miller as well as other current and former military officials at Guantanamo.
US military authorities have denied that the inmates had been tortured at Guantanamo.
Rasul was the lead plaintiff in another case that went all the way to the US Supreme Court, which ruled in June that Guantanamo detainees had the right to a lawyer and access to the US justice system.
Stress-inducing techniques were
used, according to the complaint
After their release, three of the plaintiffs from Tipton, in central England, complained of poor treatment, describing similar humiliation and stress-inducing techniques used in Iraq's Abu Ghraib prison.
The CCR said "regular abuses described by the men include the practice of shackling prisoners hands and feet to a bolt in the floor while they were seated in a chair and while strobe lights and loud music were played in a room chilled by air conditioning set at maximum levels".
"The men were subjected to these conditions for up to 14 hours at a time."
The CCR said the action was being brought under the US Alien Tort Claims Act, the Fifth and Eighth Amendments to the US Constitution, the Geneva Conventions, and Religious Freedom Restoration Act.