Mahathir, who fired his then-deputy Anwar in September 1998, earlier this month reiterated that he had considered Anwar unfit for office because it was wrong to "have a sodomiser" in the cabinet.
Anwar said he was "shocked" by Mahathir's remarks, especially after Anwar won $1.2 million in a libel suit last month against an author who had claimed Anwar had extramarital and homosexual affairs.
Homosexual acts are illegal in mostly Muslim Malaysia. "Despite the recent court ruling on the accusation of sexual misdemeanor ... as pure fabrication, Mahathir has no qualms in repeating [the accusation) and continuing to defame me," he said.
Many had expected Anwar to succeed Mahathir as prime minister.
"I will not allow this lie and slander to continue," Anwar said in a written statement. "I have instructed my counsel to initiate legal action."
Mahathir, speaking on the sidelines of a human-rights conference on 9 September, had said: "Imagine a gay PM ... . Nobody will be safe."
Mahathir fired Anwar from the
deputy PM's post in 1998
Anwar's lawyer, Sankara Nair, said he would send Mahathir a letter this week demanding a retraction, apology and damages of $26.7 million.
Anwar plans to sue if Mahathir fails to respond after two weeks, Nair added.
Such a suit would be similar to one Anwar filed against Mahathir for allegedly slandering him at a September 1998 news conference, where Mahathir graphically described Anwar's alleged acts of sodomy.
The High Court dismissed the suit in 1999, saying it was a frivolous abuse of the legal process.
"I will not allow this lie and slander to continue"
Dissident Malaysian politician
On Tuesday, Anwar accused Mahathir - who retired in October 2003 after 22 years in power - of continuing to make "baseless accusations to divert attention (from) his misguided policies that have harmed and caused severe damage to Malaysia".
After he was fired, Anwar was detained, tried for alleged corruption and sodomy and eventually sentenced to a total of 15 years before a successful appeal against the sodomy charge set him free last year.
He had already served a corruption sentence.