The US Centre for Constitutional Rights (CCR) and four Iraqis who say they were abused by American soldiers at Iraq's Abu Ghraib prison had filed a criminal complaint with German Federal Prosecutor Kay Nehm in November.
They were seeking to take advantage of a 2002 German law allowing for the prosecution of human rights abuses and war crimes regardless of the where they occur.
The complaint said that Rumsfeld, former CIA Director George
Tenet, a senior defence official and seven US military officers, including the former top US commander in Iraq, Lieutenant General Ricardo Sanchez, were ultimately responsible for the torture and humiliation of Iraqis at Abu Ghraib.
The German federal prosecutor said in a statement on Thursday it was up to the US in the first instance to pursue legal action against the alleged perpetrators and their superiors.
German prosecutors could only step in if US authorities failed to act, for which there was currently no evidence, the prosecutor added.
Photographs of US soldiers tormenting naked detainees sent shockwaves around the world when they emerged in April, prompting claims that policies adopted for the so-called "war on terror" created an environment that allowed the breach of human rights enshrined in US and international law.
The CCR said its action was a last resort after the failure of the US congress to properly investigate Abu Ghraib.