Authorities in the former British colony started a programme in 2003 to resume teaching mathematics and science in English, believing that poor English-language skills were undermining Malaysian students.
But Malay teachers and linguists have said that the policy has come at the expense of modernising Malay in the scientific lexicon.
Hadi Awang, president of Pan-Malaysia Islamic party (PAS), a conservative Islamic opposition party, was among those taking part in the rally, Hatta said.
At least 124 people were detained during the police operation against the protesters, the Star online news service reported.
Musa Hassan, inspector general of police, said the organisers had greed to only send representatives into the palace after afternoon prayers but PAS and Keadilan, another opposition party, turned it into a rally, he said.
"We had no choice but to use tear gas to disperse the crowd who refused to listen to police warnings," Musa was quoted as saying by the Star online news service.
About 60 per cent of Malaysia's 27 million-strong population is Malay, while 26 per cent is ethnic Chinese and eight per cent ethnic Indian.
Najib Razak, the deputy prime minister, said on Friday that English was important if Malaysia is to compete on the global stage.
But he said that the government would ensure that Malay was the official teaching language.
"Mastering foreign languages does not necessarily erode an individual's identity," he told Bernama news agency.