"As we strive to secure a decisive victory for our organisation in the upcoming elections, we must remember our primary task. It is to eradicate poverty and ensure a better life for all," the 90-year-old ex-president said.
It was only the second campaign appearance for Mandela, who wore a shirt in the governing party's green, gold and black colours emblazoned with the campaign slogan "Together we can do more".
Zuma followed with a message that South Africa belongs to both blacks and whites, 15 years after the fall of white-minority apartheid.
The ANC and the breakaway Congress of the People (Cope) party were both holding their final campaign rallies on Sunday, with the ruling party facing its most serious challenge since the end of apartheid in 1994.
"We reaffirm that South Africa belongs to all of us, black and white," Zuma said.
"Working together we will ensure that no South African ever feels they are less valued than others because of their race, culture or religion."
With the ANC expected to dominate Wednesday's vote, Zuma is set to become president after the elections, but Cope and a revitalised Democratic Alliance are expected to make enough of an impact to reduce its current two-thirds majority in parliament.
Such a loss would prevent the ANC from pushing through constitutional amendments.
"The 2009 election is indeed a defining moment for the ANC and the country. Only a few months ago pessimistic predictions were made about the ANC," Zuma said.
The ANC is facing criticism over its performance in tackling poverty, crime and Aids.
The ANC has promised to do more to bring economically disadvantaged blacks into the mainstream economy through land reform and affirmative action programmes, but it could prove to be a daunting task since Africa's biggest economy is on the brink of recession.
Cope, which was formed in September after the ANC forced Thabo Mbeki, the then-president, to resign, could garner about 20 per cent of the vote.