Scores of foreign dignitaries congregated in the heavily fortified presidential palace in Kabul to see Karzai place his hand on the Quran and take the oath of office.
Despite Taliban threats of violence, the capital was quiet during the inauguration.
Karzai, 47, was first picked by Washington to lead an interim government after US and Afghan forces drove the Taliban from power in late 2001, following the movement's refusal to surrender al-Qaida leader Usama bin Ladin in the wake of the September 11 attacks on the US.
Karzai also swore in his two deputies, Ahmad Sha Masud and Abd al-Karim Khalili, members of the country's two largest ethnic minorities.
Wary of attacks by Taliban or al-Qaida fighters, Afghan and international forces launched their biggest security operation since the 9 October election that gave Karzai a landslide victory.
Police sealed off the 4km route from Kabul's airport to the palace, and Nato troops patrolled the city on foot.
Before the ceremony, Karzai thanked the United States, his main sponsor, for its help.
Rumsfeld (R) said the US military
mission was not over
"Without that help, Afghanistan would be in the hands of terrorists," he said.
"Terrorism as a force is gone," he said, but added: "As individuals they are all around and we will continue to look for them."
The list of dignitaries included Rumsfeld, Iranian Foreign Minister Kamal Kharrazi and Pakistan's Interior Minister Aftab Khan Sherpao. Al-Akhdar al-Ibrahimi, special adviser to UN Secretary-General Kofi Annan, represented the world body.
Rumsfeld cautioned that the military mission was not over.
"There are still groups, extremists, that would like to take this country back - the Taliban, the al-Qaida - and use it for a base for terrorist activities around the world as they did on 9/11," he told a group of special forces soldiers at Bagram. "But it's not going to happen."
Annan warned in a report to the Security Council that unless Karzai tackles Afghanistan's surge in opium production and its arms proliferation, much of its recent progress could be seriously undermined and "the economy may well be subsumed by the illicit drugs industry".
Karzai faces daunting challenges ahead in his five-year term.
Opium poppy cultivation jumped
more than 60% this year
Fighters continue to harass US and Afghan forces across a broad swathe of the south and east. American officials expect to keep their force strength at about 18,000 at least until after parliamentary elections scheduled for the spring.
However, Karzai has said the drug economy, which now accounts for an estimated one-third of national income, is a bigger threat than the fighters and will be the top priority for the coming years.
UN surveys show cultivation of the opium poppy in Afghanistan, from which most of the world's heroin is refined, has jumped more than 60% this year, and warn that drug-smuggling mafias are taking an iron grip on the country.