"From today, none of the lawyers, except Iraqi lawyer Khalil Dulaimi, will have the right to act on behalf of Saddam," read a statement from the family, signed by Saddam's daughter Raghad.
"They used their position to further interests not linked to the case."
Saddam, who was ousted in April 2003 after the US-led invasion of Iraq and captured the following December, is in US custody near Baghdad, awaiting trial on charges of crimes against humanity.
No trial date has been set.
A person close to the family with intimate knowledge of the case, who spoke on condition of anonymity so as not to hurt relations with the family, said Raghad and other family members were upset by statements issued by various lawyers and wanted only one legal voice to speak on Saddam's behalf.
The family did not say what statements had upset its members.
Saddam's daughter, Raghad,
issued the family statement
However Saddam's former chief lawyer, Jordanian Ziad al-Khasawneh, who resigned on 7 July, claimed members of the legal team, especially Americans such as former US attorney general Ramsey Clark, were critical of his statements rebuking the American occupation of Iraq and declaring the resistance as "legitimate".
He claimed Clark advised Raghad and other members of Saddam's family that such statements hurt Saddam's defence.
The source added that the many subsequent powers of attorney issued by Saddam's legal team to other Arab and international lawyers also confused the family.
The source dismissed speculation that the legal team may in the future be made up mainly of foreign lawyers.
When he resigned, al-Khasawneh accused the family of trying to give foreign lawyers, mainly Americans, total control of the defence team, and sideline the Arabs.
Monday's statement left the door open for future appointments.
Saddam's first charges relate to
the 1982 killing of 142 villagers
"Any lawyer who would later be invited by the family to join the defence committee will be explicitly authorised by the family to make statements in due time," the family's statement said, adding "all powers of legal representation made by any member of the family or by [Saddam's legal team] to any lawyer or any other person are now deemed cancelled".
Saddam's legal team included 1500 volunteers - mainly Arabs - and 22 lead lawyers from several countries including the United States, France, Jordan, Iraq and Libya.
Prominent among them was Libyan law professor Aicha Moammar al-Gadhafi, daughter of the Libyan leader, and Clark.
No lawyer was at Saddam's side when he was arraigned in July 2004 in Baghdad on broad charges that include killing rival politicians over a 30-year period; gassing Kurds in Halabja in 1988; invading Kuwait in 1990; and suppressing Kurdish and Shia uprisings in 1991.
But the Iraqi Special Tribunal has allowed al-Dulaimi, the Iraqi member of the defence team, to meet Saddam at least four times this year, including twice when Saddam was being questioned.
Saddam is expected to stand trial in September in the first of several anticipated trials for the former leader and his chief lieutenants.
Last month, the court filed the first charges against Saddam over the 1982 killing of 143 residents of the village of Dujail, northeast of Baghdad, where he had been the target of a failed assassination attempt.