Wijdan Michael, Iraq's human rights minister and one of those staying away, said: "We are still in the government but we are boycotting cabinet meetings."
The withdrawal comes at a time when the prime minister is under growing pressure from frustrated US officials to make demonstrable progress in reconciling Iraq's warring factions.
In a separate development, US diplomats held the first meeting of a new security committee in Baghdad with officials from Iran.
Establishing the security sub-committee has been the main achievement so far of new face-to-face contacts between Washington and Tehran - who have had no diplomatic ties for almost 30 years.
Sean McCormack, a US state department spokesman, said: "It is an established channel of communication and we will see in the future as to whether or not it is a useful channel of communication."
'Frank and serious'
A US embassy official called the first meeting, which lasted four hours, "frank and serious".
After it ended, Ryan Crocker, the US ambassador to Iraq, met Hassan Kazemi-Qomi, his Iranian counterpart, for two hours of further talks in the office of Mowaffak al-Rubaie, an Iraqi security adviser.
The US accuses Iran of fomenting unrest in Iraq by supporting Shia fighters and supplying weapons such as armour-piercing bombs used to kill US troops.
Iran denies it is responsible for violence and blames the US for unleashing sectarian strife after its 2003 invasion.
A US military spokesman said on Monday that Iran had trained some of the Shia fighters who were behind more than 70 per cent of attacks on US troops in Baghdad last month.
The two countries also have long-running feuds over other issues such as Iran's nuclear programme, but officials say they were not raised in the Iraq talks.
Also on Monday, a suicide truck bomber killed at least 28 people in the northern city of Tal Afar, including 19 children.
The morning blast caused houses to collapse and officials said the death toll could rise.
Residents said children were playing hopscotch and marbles outside the houses at the time of the explosion.
"This is an ugly crime. I cannot understand how the insurgents did not think about these children," Kahlil Atta, a wedding photographer in the city, said.
US soldiers dead
The US military also reported five new American deaths on Monday.
Four soldiers were killed in a combat explosion in Diyala province north of Baghdad on Monday, a day after a soldier was killed during fighting in the capital.
Sixty bodies were also found in a mainly Sunni area that had been under the control of al-Qaeda in Iraq west of Baquba, according to a Diyala police official.
The US military said it had no information about the discovery.
At least 53 other people were killed or found dead elsewhere in Iraq, according to police officials.