"We will not return to the court unless the judge is sacked," said Jordanian Saleh al-Armuti, one of the four lawyers in the team, on Tuesday.
In chaotic scenes on Sunday, Saddam's lawyers staged a walkout from the court during a first hearing chaired by a new chief judge, Rauf Abd al-Rahman.
Armuti said the lawyers, who all left Baghdad on Monday, would not attend the next hearing scheduled for Wednesday "because the court was aggressive with our clients and the laywers".
"We made several requests to the judge which he turned down, notably a written request for the court to be transferred to Jordan or Qatar," said the lawyer.
He slammed what he called the judge's "cavalier attitude" and said his appointment had been "illegal" because it came from a government whose authority had run out.
Abd al-Rahman, adopting a tough line in contrast to his predecessor, made it clear from the very start of Sunday's hearing he was not going to tolerate the outbursts by defendants that had marked previous hearings.
But his stand resulted in the walkout of the entire defence team and half of the defendants, including Saddam.
As the defence team left the court, the judge said they would not be allowed to return, but court spokesman Raed al-Juhi said later there were procedures to bring them back.