Anwar's counsel, Sankara Nair, filed the suit at the Kuala Lumpur High Court on Friday.
"We are seeking damages and an injunction to prevent Mahathir from making similar allegations in the future," Nair said, adding they would let the court decide on the compensation.
The suit will set the stage for yet another legal battle between the two who were once close friends and colleagues but became bitter foes after a fallout eight years ago.
Mahathir, who retired in October 2003, reiterated late last year that he fired Anwar as his deputy in 1998 because his alleged sexual orientation made him unfit for office.
"I cannot have a sodomizer in my Cabinet ... Imagine a gay PM ... nobody will be safe," he told reporters in September.
Sodomy is a crime in the conservative, mostly Muslim Southeast Asian country.
"I cannot have a sodomizer in my Cabinet ... Imagine a gay P.M. ... nobody will be safe"
Ex-prime minister of Malaysia
Anwar, in his suit, accused Mahathir of speaking "falsely and maliciously" to tarnish his character.
He said his reputation had been "gravely injured" as the comments had been widely published by local and international media.
Anwar had earlier demanded a written apology and damages totalling $26.7 million from Mahathir, who remained defiant and pledged to defend himself in court.
Mahathir's aides were not immediately available for comment.
Anwar was deputy premier under Mahathir when their relationship began to crack in early 1998 after Anwar became vocal about government corruption amid other policy
Mahathir fired Anwar in September of that year, accusing him of being a homosexual.
Anwar accused Mahathir of
Anwar was subsequently arrested, tried for alleged corruption and sodomy, and sentenced to a total of 15 years in prison.
A successful appeal in the Federal Court against the sodomy charge set him free last year.
He had already served a separate sentence for corruption.
Anwar had earlier filed a slander suit against Mahathir in 1998, when Mahathir described Anwar's alleged homosexual acts at a news conference.
The High Court dismissed the suit in 1999, saying it was a frivolous abuse of the legal process.