The government also warned that no such body would be allowed anywhere in the country.
The decision, taken late on Wednesday, sends a clear message to Muslim purists that the government will not allow the country's image as a progressive and tolerant Muslim nation to be dented.
Abdullah Ahmad Badawi, the prime minister, told his cabinet colleagues that no group has the right to spy on people.
Radzi Sheikh Ahmed, a minister, said: "This involves invading people's privacy".
Radzi said ministers were shocked that such a squad, dubbed "snoop team" by the local media, was formed "right under their noses" by the Federal Territory Islamic Department in the administrative capital, Putrajaya.
The department had announced the formation of the 75-member Islamic Council Volunteer Squad on Tuesday, saying it will be on the lookout for offenders like Muslim couples holding hands in public.
The team aimed to patrol Putrajaya's parks and other public areas, and alert the department's enforcement unit if they spotted offenders.
The volunteers were not empowered to arrest anyone.
"For one, these people are not trained. You can't simply give any Tom, Dick and Harry the powers to spy and catch people"
Mohammed Nazri Aziz, Malaysian Minister
More than half of Malaysia's 26 million people are ethnic Malay Muslims.
There are also many ethnic Chinese and Indians who practise religions including Buddhism, Christianity and Hinduism.
Mohammed Nazri Aziz, another minister, said the government had rejected a plan by authorities in southern Malacca state to set up a similar force last year.
"For one, these people are not trained. You can't simply give any Tom, Dick and Harry the powers to spy and catch people," he said, adding that police are more than capable of enforcing the laws.
The voluntary force had evoked widespread protests from government officials, lawyers and civil society groups.
Ivy Josiah, a women's rights group leader, said religious authorities concerned about teaching morals can start with children in schools.