While taking the oath of office, Zuma promised to "protect and promote the rights of all South Africans".
Besides Mandela, two of Zuma's other predecessors - Thabo Mbeki and Kgalema Motlanthe - were at the ceremony alongside about 5,000 other guests, including 30 visiting leaders.
Tens of thousands of supporters gathered in Pretoria for the occasion.
Al Jazeera's Haru Mutasa, reporting from Pretoria, said: "It was very exciting for the 30,000 people on the lawns outside the Union Buildings.
"There was a gun salute, a parade and four or five planes flew over the Union Buildings," she said.
"The speech he gave was very reconciliatory. He was very serious, more down-to-earth."
Zuma won a mandate to rule after he led his African National Congress (ANC) party to a landslide election victory on April 22.
He will now have to deal with issues such as crime, Aids and poverty that have festered for years in South Africa.
At the top of his agenda will be navigating Africa's biggest economy through what could be its first recession in 17 years.
Improving the lives of the impoverished black majority has been an ANC policy promise since the party took power in the first all-race polls in 1994.
|Mandela was at the ceremony in Pretoria [AFP]
However, it remains an elusive goal, hampered by a worsening economy.
Zuma will have to manage the expectations of free-market capitalists - who have typically seen their idioms take prominence in South Africa - and labour and communist allies who propose increasing fiscal spending to help those living in poverty.
Zuma ran on a platform of "hope" - a surprise to many considering his image as a politician surrounded by cronies and with past accusations of corruption - and has promised to accelerate provision of housing, clinics, schools, running water and electricity.
But as the economy shrinks and job losses increase, government tax revenues will diminish.
About 208,000 people lost their jobs between the last quarter of 2008 and first quarter of 2009, with gross domestic product falling by 1.8 per cent in last year's final quarter.
"They've manoeuvred themselves into a tight corner," said Richard Downing, an economist for the South Africa Chamber of Commerce and Industry.
But he said the party's promise to improve government efficiency could succeed in helping the country by improving service delivery, particularly for firms, at low cost.
"It could make a vast difference in how the economy is run ... and it could raise the spirit of the people," he said.
Zuma - who spent 10 years in prison under apartheid - will make his cabinet decisions on Sunday.
Trevor Manuel, the finance minister credited internationally for increasing economic growth via free-market policies, is expected to be a prominent member of the new cabinet.
The health and education posts are also likely to be keenly watched, with the past administration being viewed as inept in dealing with the country's Aids crisis and education being seen as a means of blacks reversing inequalities imposed by apartheid rule.
There are more than five million people living with HIV and Aids among South Africa's almost 50 million-strong population.