The ruling, however, was suspended for one year to allow parliament to amend the law, as requested by government lawyers.
The controversial security certificates, which permit secret court hearings, undisclosed evidence and indefinite incarceration, have been enshrined in Canada's immigration act since 1978.
After the September 11, 2001, attacks on the US, they were used to jail five suspected terrorists, including the appellants - Moroccan Adil Charkaoui, Algerian Mohamed Harkat and Syrian Hassan Almrei.
The appellants' lawyers asked the court to "strike down" the law because the certificates circumvented normal judicial process and were unconstitutional, as well as "discriminatory".
Paul Copeland, Harkat's lawyer, said in June 2006 the government's refusal to divulge secret evidence in such cases, much of it gleaned from foreign intelligence sources, "emasculated" defenc lawyers.
Government lawyers argued the unusual secrecy in the cases prevented possible disclosure of intelligence and spy techniques to terrorists abroad. The court ruled this secrecy violated the suspects' constitutional right to a fair hearing.
Further, they found the slow and secretive process of determining whether security certificates are reasonable "cannot be justified as minimal impairments" of a person's right to a fair and speedy judicial hearing.
The lengthy review process "violates the guarantee against arbitrary detention," the justices wrote.
Harkat and Charkaoui were previously released on strict bail conditions, which include electronic monitoring, after several years in a Canadian prison.
Hassan Almrei and another man, Egyptian-born Mahmoud Jaballah, remain in custody.
They have been on a hunger strike for more than two months to protest their conditions at a Kingston, Ontario, detention centre.
A fifth detainee, Egyptian Mohammad Mahjoub, was ordered released last week.
Canadian politicians said the measures were absolutely necessary to thwart possible terrorist attacks. But critics argued that they breached civil liberties.
The Conservative government, which is unhappy with what it sees as a judicial system too soft on crime, said on Friday it was pleased the court had upheld the principle of the certificates.
"I am optimistic that we'll be able to put these changes in place," Stockwell Day, the public safety minister, said.
Earlier this week, the government had criticised the opposition Liberals over their refusal to renew some provisions of the anti-terror laws.
"At a time when the opposition parties are being soft on security and soft on terrorism, Canada's ... government remains unwavering in its determination to safeguard national security," Day said in a statement.