"He was considered as a threat to public security in Malaysia because he was part of Jemaah Islamiyah, trying to establish an Islamic government within the region," Syed Hamid.
"I think after holding him for so long, he can be brought back into society but at the same time we will follow closely everyone that may have ideology [of] militancy or extremism."
The JI group, allegedly linked to the al-Qaeda network, has been blamed for a series of bombing attacks around Southeast Asia, including the nightclub bombings in Bali, Indonesia, that killed 202 people in October 2002.
Yazid, a former army captain, allegedly let several senior al-Qaeda operatives use his apartment for meetings in Malaysia, including two eventual hijackers in the September 11 suicide attacks on the World Trade Centre.
The United States' 9/11 Commission report says Yazid, a US-trained biochemist, would "spend several months attempting to cultivate anthrax for al-Qaeda" in a laboratory he helped set up near Kandahar airport.
The US had sought Yazid's extradition after he was detained.
Nalini Elumalai, an activist with human rights group Suaram, said the three men were released on condition that they remain within their home districts and report regularly to police.
Security detainees undergo intensive counselling while in detention and will be freed if authorities are convinced that they have been "rehabilitated".
Details about four other men also freed from Kamunting and their alleged wrongdoing were not immediately available, although authorities linked two of them to violence in southern Thailand.
Earlier Singapore's Straits Times newspaper said that as many as a dozen people linked to JI had been released from detention under Malaysia's Internal Security Act, which allows for indefinite detention without trial.
The report, quoting a Malaysian official, said a government advisory board at the Kamunting detention centre decided that the Indonesian, Malaysian and Filipino detainees were no longer security threats.
The Straits Times quoting Malaysian government sources said there were another 30 suspected JI operatives including foreign nationals still detained at the Kamunting center.
Human-rights groups have criticised the ISA detentions, saying the suspects should be freed or brought to trial.