Wijdan Michael, Iraq's human rights minister and one of those staying away, said: "We are still in the government but we are boycotting cabinet meetings."
The withdrawal comes at a time when the prime minister is under growing pressure from frustrated US officials to make demonstrable progress in reconciling Iraq's warring factions.
In a separate development, US diplomats held the first meeting of a new security committee in Baghdad with officials from Iran.
Establishing the security sub-committee has been the main achievement so far of new face-to-face contacts between Washington and Tehran - who have had no diplomatic ties for almost 30 years.
Sean McCormack, a US state department spokesman, said: "It is an established channel of communication and we will see in the future as to whether or not it is a useful channel of communication."
A US embassy official called the first meeting, which lasted four hours, "frank and serious".
After it ended, Ryan Crocker, the US ambassador to Iraq, met Hassan Kazemi-Qomi, his Iranian counterpart, for two hours of further talks in the office of Mowaffak al-Rubaie, an Iraqi security adviser.
The US accuses Iran of fomenting unrest in Iraq by supporting Shia fighters and supplying weapons such as armour-piercing bombs used to kill US troops.
Iran denies it is responsible for violence and blames the US for unleashing sectarian strife after its 2003 invasion.
A US military spokesman said on Monday that Iran had trained some of the Shia fighters who were behind more than 70 per cent of attacks on US troops in Baghdad last month.
The two countries also have long-running feuds over other issues such as Iran's nuclear programme, but officials say they were not raised in the Iraq talks.