The authors of the US report said Malaysia failed to prosecute and punish traffickers, to provide adequate shelters and services to victims, or to afford protection from "involuntary servitude" - effectively slave labour.
The government was also cited for not prosecuting traffickers already detained under preventive laws.
"The Malaysian government needs to demonstrate stronger political will to tackle Malaysia's significant forced labour and sex trafficking problems," the report said.
Malaysian government officials contacted by Al Jazeera on Wednesday were unavailable for comment on the report's findings.
The Southeast Asian country was previously on the Tier 2 watch list meant for close scrutiny, along with India, Mexico and Russia.
Nations that meet what the state department says are minimum standards are placed under Tier 1 while Tier 2 is for those making significant efforts to do so.
The report said Malaysia had yet to establish a government-run shelter for foreign trafficking victims despite public announcements in 2004 and 2006.
"Without procedures for the identification of victims, the government continued to treat some trafficking victims as illegal immigrants, and arrest, incarcerate, and deport them," it added.
Malaysia has passed the Anti-Trafficking in Persons Act this year giving it a significant potential tool to effect anti-trafficking reforms.
But the report said Malaysia still needed to show "a serious increase in efforts to punish trafficking crimes and to identify and protect trafficking victims over the coming year."
Malaysia also has yet to ratify the United Nations Trafficking in Persons Protocol 2000.
The report said women and girls mainly from Indonesia, Thailand, the Philippines, Cambodia, Vietnam, Myanmar and China were trafficked into the sex trade.
Malaysia was also accused of subjecting economic migrants to conditions of forced labour as domestic workers, or in the agricultural, construction or industrial sectors.
In 2006, a memorandum of understanding with Indonesia authorised Malaysian employers to hold the passports of Indonesian women hired as domestic help, a practice which the report says can lead to involuntary servitude.