Mullen's comments come as Barack Obama, the US president, considers whether to deploy more troops to Afghanistan.
General Stanley McChrystal, the senior commander of US and Nato in Afghanistan, is reported to be seeking about 40,000 extra US forces on top of a US troop presence that is already set to reach 68,000 by the end of the year.
But Mullen said on Wednesday that sending more US troops must be predicated on the Kabul government earning credibility among Afghans.
"If we don't get a level of legitimacy and governance, then all the troops in the world aren't going to make any difference," he said.
"You have to have a governance, not just in Kabul ... but also in provinces, in districts ... the legitimacy really needs to be in the eyes of the Afghan people."
Karzai was declared the winner of the Afghan presidential election on Monday by the country's Independent Election Commission (IEC), after Abdullah Abdullah, Karzai's rival in a runoff vote, pulled out.
Abdullah, who withdrew after saying that the poll was not free and fair, said on Wednesday that the IEC decision to name Karzai the winner of the election was illegal.
"This decision does not have any legal basis and a government which takes power based on such a commission's decision cannot have legitimacy," Abdullah said on Wednesday in his first public appearance since the announcement.
"Such a government which lacks legitimacy cannot fight corruption.
"A government which comes to power without the people's support cannot fight phenomena of terrorism threats, unemployment, poverty and hundreds of other problems."
The discovery of widespread fraud in the first round on August 20 resulted in Karzai's share of the vote falling below the 50 per cent threshold needed to avoid a runoff.
Ballot-box stuffing and distorted tallies led to more than one million votes in favour of the incumbent being thrown out.
The validity of the electoral process and the independence of the IEC have both been called into question.
Abdullah cited the government's refusal to accept his demands for changes to the IEC for his decision to boycott the runoff.
Al Jazeera's James Bays, reporting from the capital, Kabul, said Abdullah has shown he is not going to accept losing the election.
"He said the Independent Election Commission proved itself not independent throughout this process," our correspondent reported.