The American Civil Liberties Union (ACLU) and Human Rights First on Tuesday filed suit in federal district court in Rumsfeld's home state of Illinois on behalf of eight former detainees who said they were severely tortured.
All eight were subsequently released without being charged.
"Secretary Rumsfeld bears direct and ultimate responsibility for this descent into horror by personally authorising unlawful interrogation techniques and by abdicating his legal duty to stop torture," said Lucas Guttentag, lead counsel in the case.
The Pentagon denied that it ever sanctioned or condoned the abuse of detainees.
"There have been multiple investigations into the various aspects of detainee abuse. None have concluded there was a policy of abuse," the Defence Department said in a statement.
The ACLU filed similar complaints against three other senior officers: Colonel Thomas Pappas, General Janis Karpinski and Lieutenant-General Ricardo Sanchez on behalf of prisoners mistreated at Iraq's Abu Ghraib prison.
Karpinski (L) and Rumsfeld (C)
are accused in the suit
The suit against Rumsfeld focuses on an order he signed in December 2002 that authorised new interrogation techniques for detainees in the "war on terror" being held at Guantanamo Bay in Cuba.
These included "stress positions", hooding, 20-hour interrogations, removal of clothing, exploiting phobias, prolonged isolation and sensory deprivation.
When evidence became overwhelming that prisoners were being tortured, Rumsfeld turned a blind eye, the suit alleges.
"Secretary Rumsfeld knew full well that his orders were causing torture and he knew that torture was occurring on a widespread basis and he did not stop it," Guttentag said.
The plaintiffs want the court to declare Rumsfeld's actions unconstitutional and a violation of US and international law.
They are seeking monetary damages for their injuries and all eight are willing to come to the US to testify.
The plaintiffs - four Afghan citizens and four Iraqis - allege treatment that included beatings, being cut with knives, sexual abuse and humiliation, being locked in coffin-like boxes, being deprived of food and water and threatened with execution and hung upside down for hours.
"Secretary Rumsfeld bears direct and ultimate responsibility for this descent into horror by personally authorising unlawful interrogation techniques and by abdicating his legal duty to stop torture"
Lead counsel, Lucas Guttentag
Arkan Muhammad Ali, a 26-year-old Iraqi held for a year from June 2003 to 2004, alleges that US personnel twice beat him unconscious, used a knife to repeatedly stab and slice his forearm, burned and shocked him with a metal device, locked him naked for several days in a small wooden box, urinated on him and made death threats against him.
Mehboob Ahmad, a 35-year-old Afghan citizen held for five months in 2003, said he was probed anally, hung upside down from the ceiling by a chain and hung by his arms for extended periods.
The mistreatment of prisoners became an international scandal after the appearance last year of pictures showing sexual abuse of men at Abu Ghraib.