The UN human rights office said on Friday that the strikes hit three families who were staying in a makeshift straw house after being displaced from their homes three months ago by fighting.
The attack on Tuesday in the Mawza district of the southwestern province of Taiz killed at least 20 people, including four children, according to residents.
The office said in a statement that the strike was carried out by “Arab Coalition Forces”, referring to the Saudi-led alliance fighting Houthi rebels in support of Yemen’s internationally recognised government.
“There do not appear to have been any military objectives anywhere in the immediate vicinity of the destroyed house,” it added.
The Saudi-led coalition has faced repeated criticism over civilian casualties in Yemen. It accuses the Houthi rebels of using civilians as human shields.
The coalition has admitted responsibility for some strikes that killed non-combatants, including a raid on the rebel-held capital Sanaa in October 2016 that killed more than 140 people at a funeral ceremony.
The United Nations called for a “comprehensive and impartial investigation” into the latest incident.
The conflict in Yemen has escalated dramatically since March 2015, when the Saudi-led forces launched a military operation against the rebels.
Since the conflict began, more than 10,000 people have been killed and millions have been driven from their homes.
The country is also facing a health crisis, with the charity Oxfam reporting 360,000 suspected cases of cholera in the three months since the outbreak started in April.