The complaint is being filed on behalf of 11 Iraqis who were detained at Abu Ghraib and one Saudi who was held at Guantanamo Bay.
The CCR says they were beaten, deprived of sleep and food and sexually abused.
'No safe haven'
Although a similar attempt to prosecute Rumsfeld in Germany two years ago failed, the German lawyer representing the detainees said he was confident that this time it would succeed.
"We failed two years ago because there was an ongoing investigation in the United States, but it is now clear that there is no chance of prosecuting high-ranking officials in the US," Wolfgang Kaleck said.
"If we fail here, we will try in France, or in Spain. We want to show that there will be no safe haven anywhere in the world for him."
German law allows the pursuit of war crimes cases regardless of where they originate in the world.
Gita Gutierez, a US lawyer for the Saudi Guantanamo inmate named in the suit, Mohammed al Qahtani, said Rumsfeld bore ultimate responsibility for his treatment.
He was kept in isolation for 160 days, deprived of sleep for 48 days, forbidden to pray unless he co-operated with interrogators and was sexually assaulted by a female soldier, Gutierez said.
A Former US Army Brigadier, General Janis Karpinski, who ran the Abu Ghraib prison at the time photographs depicting the abuse of prisoners were widely published, said she was willing to testify against her former boss Rumsfeld.
Karpinski, who was blamed for Abu Ghraib, said the abuse was directed by military intelligence, over which she had no say.
"When I look back at it now, when I see the footage, when I see the Iraqi people I see a loss of hope in their faces.
"I see desperation in their faces and I know that we in many ways contributed to this situation."
Rumsfeld resigned last week after the Republican party lost control of the US congress in elections widely seen as a referendum on the war in Iraq.