The election was delayed for two days by arguments between the supporters of the two rival camps.
Zuma, sacked by Mbeki two years ago, was among the first to vote at the University of Limpopo in the northern city of Polokwane where the ANC is staging its five-yearly conference.
The vote will be counted manually rather than electronically at the insistence of Zuma supporters, who say they fear rigging by Mbeki.
Many investors are nervous about a Zuma victory, fearing it could tilt South Africa to the left and change Mbeki's conservative policies, which have kept the economy growing.
Zuma, who is backed by trade unions and the Communist party, has tried to woo investors by saying he will not change economic direction.
On Monday supporters of the men, formerly allies, staged rallies outside the conference, singing partisan songs and chanting slogans.Zuma lead
If Zuma wins, he will almost certainly also succeed Mbeki as the country's president in 2009 due to the party's electoral dominance.
Mbeki is barred by the country's constitution from seeking a third term as the country's president.
But if he were to remain the ANC leader, he would influence the party's choice for the next president.
Zuma took the lead over Mbeki in branch elections for the ANC presidency, winning the support of five out of the nine ANC provincial branches as well as the backing of the women's league and youth league in a first round of voting before the conference.
The infighting caused deep concern in an organisation that prided itself as presenting a united front during the struggle against apartheid and has ruled virtually unopposed since 1994.
|Rallies for both men were held outside the|
ANC conference in Polokwane [AFP]
"The organisation is going through deep strain," Smuts Ngonyama, ANC spokesman, said.
Nelson Mandela, the 89-year-old former president, who has retired from politics but who is still seen as a unifying figure in the country, also expressed his concern at the divisions in the party.
"It saddens us to see and hear of the nature of the differences currently in the organisation," he said in a message to the delegates distributed by the South Africa Broadcasting Corporation.
"Whatever decision you are to make at this conference, including decisions about leadership positions in the organisation, let the noble history of the ANC guide you."
Zuma is a controversial figure. He was acquitted of rape last year but is still under investigation for corruption.