The victory was a remarkable comeback for the former deputy head of state who was sacked by Mbeki when his financial adviser was jailed for fraud in 2005.
He then stood trial accused of raping a family friend half his age, and although he was acquitted, he was widely ridiculed for testifying he had showered after sex with his HIV-positive accuser in order to prevent infection.
The party's large majority in parliament makes Zuma almost certain to become state president when the constitution forces Mbeki to stand down in 2009.
However, he faces being charged with corruption after losing a recent court bid to have a series of search warrants declared illegal.
Authority underminedWhile Mbeki has two years left as South Africa's president, his authority has been badly undermined by the defeat.
Tokyo Sexwale, a billionaire businessman and a one-time leadership contender, said that the result showed the desire for change.
"The people have been vindicated. Democracy in South Africa has spoken and South Africans can rest assured that we are looking forward to a good cabinet," he said.
"The people wanted a change of leadership and it has been effected and I think it is good."
Zuma, a highly personable character in contrast to the intellectual Mbeki, cashed in on the growing disillusionment with the government's failure to eradicate poverty in the 13 years since the end apartheid.
"Mbeki lost for a lot of reasons [but] the main reason is the question of leadership style," Steven Friedman of the Pretoria-based Institute for Democracy in South Africa, said.
"He irritated a lot of people. Some of that you have to do when you are in power. But much of it could have been avoided."
Investor fearsAlthough Mbeki can point to an uninterrupted period of growth, unemployment is unofficially estimated to be around 40 per cent.
Despite investors' fears, Ronnie Kasrils, the intelligence minister, denied that ANC policy was likely to change under Zuma, echoing statements by other officials.
"The economy is a very central issue. That is where there will be thorough discussion ... I would not see a [policy shift] but rather finding a way of delivering to the poor," he said at the ANC conference in the northern town of Polokwane.
Union leaders welcomed the victory after throwing their weight behind Zuma and said that although they did not expect wholesale policy changes they wanted a better environment.
"We want a better environment, a respect of different voices," Zwelinzima Vavi, secretary-general of the Congress of South African Trade Unions, said.