Human Rights Watch was referring to the court in Iraq that has tried Saddam over the Dujail killings and that next week will try him for genocide against the Kurds.
"None of the Iraqi judges and lawyers has shown an understanding of international criminal law," it said in a statement on Friday.
"The court's administration has been chaotic and inadequate, making it unable to conduct a trial of this magnitude fairly.
"Based on extensive observations of the tribunal's conduct of its first trial ... Human Rights Watch believes that the Iraqi High Tribunal is presently incapable of fairly and effectively trying a genocide case."
It also questioned the extensive reliance on anonymous witnesses, which it said undercut the defence's right to confront the evidence and cross-examine their testimony.
The ousted Iraqi president, who is awaiting a verdict in his first trial for crimes against humanity, will be in the dock again on Monday for the so-called Anfal campaign of killings of tens of thousands of Kurds in the north in the late 1980s.
"Based on extensive observations of the tribunal's conduct of its first trial ... Human Rights Watch believes that the Iraqi High Tribunal is presently incapable of fairly and effectively trying a genocide case"
Human Rights Watch
The first trial of Saddam and seven co-accused in connection with the killings of 148 Shia in the 1980s was marred by the killing of three defence lawyers and the resignation of the tribunal's first chief judge over what he said was government interference. It was adjourned until October 16.
Saddam and six other defendants, including his cousin Ali Hassan al-Majid, nicknamed "Chemical Ali" for ordering gas attacks, are accused of genocide for their role in the Anfal campaign.
Kurds say tens of thousands of people perished and that entire villages were wiped out in the violence.
The Kurds see the Anfal campaign as one of the most potent symbols of their suffering under Saddam, who described Kurdish leaders as traitors.