In a speech to some 1,000 people in his home state of Kedah he also urged other Umno members and cabinet members to follow him.
Mahathir appointed Abdullah to succeed him in 2003 after serving as prime minister for 22 years.
But he remained active in politics, wielding significant influence among the party's grass-roots workers.
By late 2004 Mahathir began accusing Abdullah of nepotism, corruption and inefficiency.
Party insiders say Mahathir was angry because many big infrastructure projects he had initiated were cancelled by Abdullah.
Mahathir's decision raises the possibility of large-scale desertions by loyalists, which could split the party and bring down the government.
Umno is the dominant party in the coalition Barisan Nasional administration that has governed Malaysia since independence in 1957.