Karzai is battling to rebuild his tarnished reputation following a fraud-tainted presidential election on August 20, in which a UN-backed investigation found a third of his votes were fake.
He was named president after Abdullah Abdullah, his main rival, pulled out of a scheduled runoff vote complaining that not enough had been done to address the problems of the first round.
Karzai appeared to invite Abdullah and Ashraf Ghani, another election challenger and a former World Bank executive, into a government of national unity in Thursday's speech.
"I want to invite both of them to contribute," he said.
Abdullah told the Reuters news agency that he had no plans to accept a role in Karzai's administration.
"It's more of the same," he said.
"He has spoken in these terms, in terms of bringing changes and reform, and fighting corruption, and bringing security and reconciliation, for the last eight years, and the situation has worsened."
He also appeared to call for reconciliation with his Taliban leaders, proposing a "loya jirga", a traditional assembly which under Afghanistan's constitution can take precedence over all government institutions, including the presidency itself.
"We welcome those who are not affiliated with any terrorist organisations and whose hands are not red with Afghans' blood," he said.
The Taliban also said it would not accept his call for national unity.
"Today is not a historic day. This is a government based on nothing because of the continuing presence of foreign troops in Afghanistan," Zabiullah Mujahid, a Taliban spokesman, told The Associated Press news agency.
Among the representatives from more than 40 countries in Kabul was Hillary Clinton, the US secretary of state.
|Whole neighbourhoods of Kabul were closed off for the inauguration ceremony [AFP]
Karzai addressed many of her concerns about corruption but also struck a note of defiance after being heavily criticised by Washington and its allies since the election.
"He still raised the detention of Afghan prisoners in American jails ... he raised other uncomfortable issues for the Americans, the issue of civilian casualties for example," Al Jazeera's James Bays, reporting from Kabul, said.
"So he was able to make his speech, but it included that central element of corruption that the Americans insisted that they had to hear."
Kabul, the Afghan capital, was locked down ahead of the inauguration ceremony.
Regular flights to and from Kabul airport were cancelled for the day and a number of neighbourhoods in the capital were closed to traffic in an attempt to stop attacks.
Karzai addressed the security concerns, saying that Afghanistan's own forces aimed to assume responsibility for security in unstable parts of the country within three years.
Karzai also promised to create thousands of jobs for his impoverished people as part of a reconstruction drive in the country.
Nabi Misddaq, an Afghan scholar and writer, said that the broad nature of the issues covered in Karzai's speech showed the seriousness of the challenges he faces in his next term.
"Challenges from the international community that he must do serious things to bring bribery and all kinds of undesired things that have been going on over the past six years under control," he told Al Jazeera.
"Challenges from Afghans that he has not been a very effective and strong president. And also [the] challenge that the international community is not going to be around [in Afghanistan] for long."