Flash floods have destroyed at least 1,700 houses in Mahara and Hadramaut alone and both provinces were declared disaster zones on Friday.
The damage has been extensive in Hadramaut, also Yemen's largest province, because most homes are made of mudbrick.
Attempts to get medical supplies and relief aid to the region have been hampered as many of the roads in the area have been completely flooded.
Heritage site threatened
Among the affected areas is the Unesco world heritage site of Shibam, where the area's historic mudbrick buildings are threatened with collapse.
Shibam, which was totally isolated by the flood waters, is home to more than 20,000 people and is famous for its mudbrick high-rise buildings that have given the town the nickname "the Manhattan of the desert".
A cargo vessel also ran aground off the port of Nashtun in Mahra, but all 17 crew members were rescued.
Local authorities have reported serious damage to roads and power and water distribution networks.
Yemen is one of the poorest countries in the world and its government is already struggling to contain rebellion in the north, unrest in the south and the re-emergence of al-Qaeda in the region.
An international aid appeal has been launched.
The storm formed in the Indian Ocean this week, first headed to the Horn of Africa and the Gulf of Aden, then turned and moved towards the southern coast.