Speaking in Miami on Friday, Rumsfeld conceded that "detaining people without lawyers seems unusual" but insisted those detained were not common criminals.
"They are enemy combatants and terrorists who are being detained for acts of war against our country. And that is why different rules have to apply."
His defence comes amid growing court challenges and international criticism of the US assertion of a right to hold detainees without trial indefinitely – particularly as it is unlikely there is to be any clear ending to the "War on Terror".
The defence secretary said some detainees would be tried before military commissions "for serious crimes," others would be transferred to home countries willing to take responsibility for them, and still others would be released if considered no longer a threat.
No end soon
But he unveiled a new review process that is being put in place with the expectation that many prisoners would remain in US custody for years as "enemy combatants" without trial.
"I recognize that in our society the idea of detaining people without lawyers seems unusual, detaining people without trial seems unusual"
US defence secretary
Rumsfeld argued that the United States, which toppled the Taliban regime in Afghanistan within months of the 11 September 2001 attacks on World Trade Centre and Pentagon, was still at war with the Taliban and al-Qaida.
"The Taliban continue to wage war against a legitimate government in Afghanistan, and against our coalition forces there, by public declarations, as well as hostile acts," he said.
"Al-Qaida continues to wage war on Americans and on all civilized people with disturbing regularity. No one could possibly claim that the conflict had ended."