The defendants were top officials in the Bush administration such as Donald Rumsfeld, the former defence secretary, and 10 military commanders including retired General Richard Myers, the former chairman of the Joint Chiefs of Staff.
The justices rejected the appeal without comment on Monday.
A year ago, the supreme court sent the case back to the appeals court for further consideration in view of the high court's landmark ruling that Guantanamo prisoners have legal rights.
"This case presents a unique and compelling opportunity for this court to affirm that torture at Guantanamo was a violation of fundamental rights"
Statement from former Guantanamo inmates
But the appeals court for a second time dismissed the action, ruling that the detainees had no legal basis for their claims at the time they were at Guantanamo, that Pentagon officials were entitled to immunity and that the religious right law did not apply to the prisoners.
The Obama administration opposed the high court review of the case, saying the appeals court's decision was correct and that a further review of the case was unwarranted.
"This case presents a unique and compelling opportunity for this court to affirm that torture at Guantanamo was a violation of fundamental rights," the four former inmates said in their appeal.
"Left in place, the court of appeals' decision will be a final assertion of judicial indifference in the face of calculated torture and humiliation of Muslims in their religion. The decision cannot stand."
Rasul, Iqbal and Ahmed allege they travelled to Afghanistan from Pakistan to provide humanitarian relief the month after the September 11, 2001, attacks against the US, while al-Harith says he travelled to Pakistan the same month to attend a religious retreat.