The group, including nine women and 16 children, sought refuge on Tuesday in several mosques in the northeastern state of Kelantan which borders Thailand, said state police chief Zulkifli Abdullah.
"They came in without travel documents and we have handed them over to the immigration department for further action," he told journalists.
Zulkifli said the Thais had told police that they feared for their safety in their home provinces, where a Muslim insurgency is raging. Nearly 900 people have died in the Thai south over the past 20 months.
"They are just worried about their safety but as far as police are concerned, we are worried about them coming in without documents," he said.
One of those detained, identified as Salleh, told the official Bernama news agency that they fled after the army entered their village in Thailand's Narathiwat province two days ago and a village chief was shot dead.
Malaysia's Foreign Minister Syed Hamid Albar said the country was prepared to provide temporary shelter to the Thai illegals if the situation continued to be unstable.
He said Malaysia was always sensitive to the problems of the people in Thailand and enjoyed close relations with its neighbour.
"However we don't want to interfere with the internal affairs of Thailand," he was quoted as saying by Bernama, adding that Malaysia would not recognise the illegal Thais as asylum-seekers.
The insurgency in Thailand's south
has cost hundreds of lives
Malaysian police tightened security at all border points to prevent further movements. "We have our patrols working around the border so we will tighten the security around there," Zulkifli said.
News reports quoted Malaysian police as saying they feared thousands more Thai Muslims could cross the border in the next few days, but Zulkifli said he "cannot say for certain" if others were expected.
Marine police in the area said they were carrying out 24-hour patrols along Malaysia's northeastern coast near the border with Thailand.
The violence in southern Thailand has flared anew with a series of bomb blasts and shootings across the troubled region leaving three people including two policemen dead and 21 injured since late Wednesday.