The crowd later marched to the royal palace in a procession before protesters presented a memorandum to a representative of the king.
"It is a good signal that Malaysians want freedom and democracy, and they want free and fair elections"
Anwar Ibrahim, People's Justice Party
The document asked the monarch to intervene to ensure that the election system undergoes reform before general elections, which are expected to take place early next year.
About 70 non-governmental organisations and opposition parties organised the protest, demanding the removal of phantom voters from electoral rolls and an end to government workers using absentee ballots.
They also called for access to state-controlled media by all political parties and an end to vote-buying and other irregularities in electoral practice.
"This is our right. Our rulers are so proud of our democracy but in fact our democracy is worse than Burma, worse than Bangladesh," said Rosli, a 40-year-old government worker.
"We just want to correct what is wrong. We just ask for fair elections." Musa Hassan, national police chief, was quoted by the Sunday Star newspaper as saying that authorities had to fire tear gas and water cannon after protesters refused to disperse.
He said 4,000 people attended the rally, but organisers estimate at least 30,000 took part.
The rally was the largest political demonstration in Malaysia since supporters of Anwar Ibrahim, then deputy prime minister, protested for several days in September 1998.
That demonstration was in protest at Ibrahim's dismissal from the cabinet and ruling party by Mahathir Mohamad, who was then prime minister.
Anwar later founded the People's Justice Party, one of three opposition parties involved in the latest demonstration.
"It is a good signal that Malaysians want freedom and democracy, and they want free and fair elections," he said.
Anwar has called for further street protests until changes in electoral policy are implemented.
However, Najib Razak, Malaysia's deputy prime minister, said the government would not tolerate illegal assembly.