Villagers were taken by surprise by the typhoon's arrival as it had been forecast to head north, but instead turned south to unprepared areas where it triggered the worst flooding seen in 50 years, Chao said.
One Hsiao Lin survivor said from a temporary shelter in a neighbouring town that "the mountain just collapsed on us".
"I live lower down and survived, but all my relatives were buried," he said.
Efforts to search for people trapped in the storm were hampered on Tuesday when a rescue helicopter crashed in heavy fog with at least six people on board.
The six crew members were unaccounted for after the crash.
The helicopter was part of Taiwan's National Emergency rescue operation taking place in the south.
Morakot has now weakened to a tropical storm but it is still wreaking havoc on China's southeastern coast, triggering a massive landslide in Zhejiang province that toppled six apartment buildings burying an unknown number of residents late on Monday.
China's state-run Xinhua news agency said on Tuesday that the landslide in Pengxi town destroyed six four-storey buildings at the foot of a mountain at 10:30pm local time on Monday.
|Chinese state media said six survivors were pulled from the toppled Pengxi buildings
Six survivors were pulled out - one in critical condition - but rescuers did not yet know how many people were buried under the rubble and search operations were hampered by the amount of mud and rock.
The storm also sparked the evacuation of about 1.4 million people from Zhejiang and neighbouring coastal province Fujian, Chinese authorities said.
Al Jazeera's China correspondent, Tony Cheng, said many areas had been evacuated but many people were still in the storm's path and since the buildings collapsed at a late hour when people would have likely been home, casualty figures were bound to rise.
Many of the areas affected by heavy rains are mountainous and building standards are not known to be very high there, leaving many vulnerable to landslides, our correspondent said.
Francis Markus, a spokesman for the International Red Cross Federation and the Red Crescent Society in China, told Al Jazeera from Beijing that things were "still very much on edge".
He said residents who had evacuated from the area were looking for the weather and terrain to stabilise before making their way back home.