Local sources told Al Jazeera that Tuesday's attack targeted an area west of the city of Taiz, where a number of displaced families have sought safety.
Those killed are believed to have been in their homes when the attack took place, with the majority of the victims said to be from the same family.
Yemen's internationally recognised government confirmed that around 20 people were killed in the al-Atera village. It gave no further details on the attack.
Yemen has been torn apart by conflict since 2014, when Houthi rebels, allied with troops loyal to former President Ali Abdullah Saleh, captured large expanses of the country, including the capital, Sanaa.
A coalition of Arab countries assembled by Saudi Arabia launched an air campaign against the rebels in March 2015 to try to restore the internationally recognised government of President Abd-Rabbu Mansour Hadi to power.
Yemen's human rights minister, Mohammed Askar, on Tuesday called for a government investigation into what he described as an "unfortunate incident", while Houthi spokesman Mohammed Abdul-Salam condemned the attack as a "monstrous crime".
The UN refugee agency, UNHCR, said in a statement that it was "deeply shocked and saddened at reports of the deaths and injuries of a number of internally displaced persons in an aerial attack" on the area.
The statement said the civilians had fled fighting in the nearby Mokha district, on the Red Sea coast.
"This latest incident once again demonstrates the extreme dangers facing civilians in Yemen, particularly those attempting to flee violence, as they disproportionately bear the brunt of conflict," UNHCR said.
Coalition-backed forces recaptured the port of Mokha earlier this year as part of a major ongoing offensive to drive rebels from Yemen's lengthy coastline.
Tuesday's attack was the latest in a series of coalition bombings that have hit schools, hospitals, markets and private homes across northern Yemen.
Since the Saudi-led coalition began its attacks, the conflict has killed more than 10,000 people, forced millions from their homes, damaged infrastructure and pushed the country to the brink of famine, according to the UN.
The country is also facing the "world's worst cholera outbreak", according to the UN.