A Yemeni official said on Thursday that rocks had slid off a mountain late on Wednesday and collapsed into the Dhafari village - about 20km southwest of Sanaa, the capital - destroying its 27 houses.
Medics on the scene said some 20 bodies had been pulled from the rubble of where part of a mountain collapsed on the village.
The medics, who spoke on condition of anonymity as they were not authorised to talk to the media, said they fear many more bodies lie under the rubble.
Police have cordoned off the area and are searching for survivors. On Thursday, soldiers and civilians with shovels dug through the ruins of stone houses while a bulldozer pushed at larger stones.
Rashad al-Eleimi, Yemen's interior minister, has visited the site to monitor the search.
State news agency Saba said dozens of people were injured - some seriously - and have been taken to hospital.
Aljazeera's correspondent in Yemen reports that about 20 houses were destroyed in the landslide.
Community leader Saleh Hamoud said he escaped harm, but eight members of his family, including his wife and children, were missing.
The remoteness of the village and the inexperience of rescue workers was hampering the search operation, Hamoud said.
Yemeni medics fear many more
bodies are lying under the rubble
"We have little experience and limited capabilities. And what's making things worse is that it happened at night when everyone was asleep in their home," he said.
"Big rocks came tumbling down the side of the mountain and this making it difficult to reach victims because they're stuck under these big boulders."
Ambulances rushed to the village and major hospitals in the capital were put on emergency status.
For centuries, Yemenis have built their houses on the sides of mountains, sometimes carving homes out of the rockside.
Landslides are rare in Yemen, at the southern tip of the Arabian peninsula, but the country is prone to floods in spring and summer.