Most of those arrested in the months following the September 11, 2001, attacks on the World Trade Centre in New York are held at Guantanamo Bay, the US naval base in Cuba.
The groups on Thursday published the list of 39 detainees based on information gleaned from interviews with former prisoners and officials in the US, Pakistan, Afghanistan and Yemen.
Amnesty says it is not sure if the missing prisoners are in any US detention facility, claiming their report offers evidence that Washington has been lying.
"For its part, the CIA has always maintained that it operates in strict accord with American law," Sean Casey, a US State Department spokesman, said.
George Bush, the US president, had late last year acknowledged the existence of secret detention centres but did not specify any locations.
The rights groups say the men were among those captured in highly controversial "renditions" in which suspects were detained outside the US and flown to top secret American jails around the world.
In Europe, a Swiss senator heading an investigation into alleged CIA secret prisons and flights in Europe is expected to present on Friday new findings on what he calls a "spider's web" of human rights abuses.
Dick Marty, who is leading an inquiry on behalf of the Council of Europe, had spoken to former CIA agents to corroborate his earlier accusations that CIA planes landed in Poland and Romania to drop off detainees, a source familiar with the investigation said.
The European Parliament came to the same conclusion after completing its own investigation in February.
Both the Polish and Romanian governments have vehemently denied Marty's allegations.
Last year, he accused 14 European nations of colluding to help the CIA spirit terror suspects to illegal detention facilities, and said European governments "did not seem particularly eager to establish" the facts.
While details of most of the detainees remained sketchy, information for at least 21 of the detainees had been confirmed by two or more independent sources, Anne Fitzgerald, a senior adviser for Amnesty International, said.
Meanwhile in Washington on Thursday, the US Senate Judiciary Committee endorsed a bill that will restore basic legal rights to inmates of Guantanamo Bay detainees, allowing the accused to challenge their detention in a US court.
The move came days after the US administration suffered a severe legal setback in its 'war on terror' after a military tribunal dismissed charges against two Guantanamo inmates.
"Habeas corpus was recklessly undermined in last year's legislation," Senator Patrick Leahy, the Judiciary Committee chairman, said. "I hope that the new Senate will reconsider this historic error in judgment and set the matter right."
Judges who heard both cases on Monday ruled they had no jurisdiction to proceed with military commission trials as neither inmate was classified as an "unlawful enemy combatant".