In a serious setback to the so-called war on terror, the US Second Circuit Court of Appeals ruled on Thursday that only the US Congress can authorise such detentions.
The court then ordered the government to release Jose Padilla from military custody within 30 days.
Padilla is a suspect in an alleged al-Qaida plot to detonate a radioactive "dirty bomb" in the United States.
"Where... the president's power as Commander-in-Chief of the armed forces and the domestic rule of law intersect, we conclude that clear congressional authorisation is required for detentions of Americans on American soil," the court ruled.
It said the government can transfer Padilla, a US citizen who is being held incommunicado in a Navy prison, to a civilian authority that can bring criminal charges against him.
"Where... the president's power as Commander-in-Chief of the armed forces and the domestic rule of law intersect, we conclude that clear congressional authorisation is required for detentions of Americans on American soil"
US appeals court ruling
"Padilla will be entitled to the constitutional protections extended to other citizens," the court panel said.
However, immediately after the ruling the Bush administration said it would seek a stay of the court decision.
White House spokesman Scott McClellan called the court's ruling "troubling and flawed".was arrested at Chicago's O'Hare airport 18 months ago as he arrived from Pakistan.
He was transported to Manhattan federal court system where he was held as a material witness in a federal grand jury investigation of the September 11 attacks.
On 9 June, 2002, the Bush administration classified him as an "enemy combatant" and he was transferred to a Navy prison in South Carolina.
His New York lawyers sought his release as well as access to their client.
Bit federal prosecutors argued Padilla should not have access to attorneys because they said he posed a threat to national security.
They also believe defense lawyers could unwittingly be used to pass messages to al-Qaida operatives.
In recent arguments before the appeals panel, Deputy Solicitor General Paul Clement argued that after the September 11 attacks Congress gave the president the right to detain American citizens indefinitely.
Primacy of law
In his dissent, Circuit Judge Richard Wesley said, "In my view, the President as Commander in Chief has the inherent authority to thwart acts of belligerency at home or abroad that would do harm to United States citizens."
Around 600 'enemy combatants'
are detained at Guantanamo Bay
But the court disagreed, although it emphasised its ruling is limited to the case of an American citizen arrested in the United States.
It does not, however, apply to those seized on a foreign battlefield or while actively engaged in armed battle against the United States.
Padilla's challenge was supported by the American Bar Association, the nation's largest legal association, as well as human rights groups.
"The court's holding reaffirms that even in the post-September 11 world, the President of the United States is not above the law," said the Lawyers Committee for Human Rights.