Breyer's views were echoed by Justices David Souter and Ruth Bader Ginsburg. Breyer and Souter said they would have heard the case on a fast track, as the detainees had requested.
"This is a perfect example of justice delayed is justice denied. All these people ever wanted was a fair hearing"
Tom Wilner, lawyer for 39 detainees
It takes four votes among the nine judges to accept a case.
Tom Wilner, a Washington lawyer who has represented Guantanamo detainees since May 2002, said: "This is a perfect example of justice delayed is justice denied.
"All these people ever wanted was a fair hearing."
Wilner is representing 39 Guantanamo inmates to challenge their detention.
The US constitution provides for a habeas corpus review that protects people from unlawful imprisonment.
Monday's decision comes in spite of two earlier Supreme Court rulings affording the detainees legal protection.
Some of the estimated 385 prisoners have spent more than five years at the US naval base in Cuba, where military panels assess inmates to see if their detentions are justified.
None of the prisoners has been tried in a civil court.
A Columbia district appeals court stripped federal courts of their ability to hear such challenges in February.
But it now has to review the work of military panels that have found each of the detainees to be an enemy combatant, raising the possibility of the supreme court revisiting the same cases it rejected on Monday.
The decision has drawn support and criticism. Chris Dodd, a Democrat senator, called the court's refusal to hear the case "misguided" and said it highlights the urgency for congressional action.
"[Guantanamo is reserved for] "the most egregious terror suspects"
Navy Commander Jeffrey Gordon
He said basic rights must be restored to the detainees and the Geneva Conventions must be adhered to.
At the Pentagon, Navy Commander Jeffrey Gordon said Guantanamo is reserved for "the most egregious terror suspects".
However, only 10 have been charged with a crime.
Michael Ratner, president of the Centre for Constitutional Rights, said: "We're disappointed and for us this is a delay that is unconscionable."