Local disaster relief officials said one woman had been rescued from a house buried in a landslide, but her one-year-old daughter died along with her teenage uncle.
"It happened so fast ... I didn't have time to save them," the woman's husband told ERA News.
Elsewhere in Kaohsiung and nearby Tainan, three people were drowned and two others were washed away by floodwaters.
The treacherous conditions even took emergency workers by surprise with one man and woman reported missing after a police motorboat that had earlier rescued them overturned in floodwaters.
In the coastal county of Yunlin firefighters have been using ladders to rescue people trapped in flooded houses.
Rescuers have evacuated about 80 people trapped by mudslides or floods in central and southern Taiwan, where electricity and water supplies in hundreds of thousands of households were affected.
Officials said several areas in the south of Taiwan have recorded up to 1,100mm of rains in the space of just 24 hours.
Kalmaegi ravaged fields and farms, causing an estimated 111 million Taiwan dollars ($3.65m) of damage, according to government statistics.
Earlier this week Kalmegi, a Korean word meaning seagull, swept through the northern Philippines leaving two people dead according to local officials.