"Lose the people's trust, and we lose the war."
The US has come under increasing criticism from Afghanis and their leaders over the deaths of civilians during US-led attempts to tackle al-Qaeda and Taliban forces in the country.
"You cannot defeat an insurgency this way," Mullen said.
Holbrooke and Karzai are due to hold a press conference on Sunday, after their meeting a day before.
Karzai admitted before the talks in Kabul, the Afghan capital, that tensions had grown between the two governments and he had not spoken to the new administration in Washington until Holbrooke arrived in Afghanistan on Friday.
But following Saturday's talks in the presidential palace, Sultan Ahmad Baheen, a foreign ministry spokesman, was upbeat about relations between the two sides.
"The United States of America once again expressed their support for the Afghan government and its people in the fight against terrorism, [support for] development, improving democracy and freedom of human rights," he said.
"And both sides emphasised greater focus in the fight against terrorism and improvement of regional co-operation."
Karzai reasserted his displeasure at Afghan civilian deaths due to US-led operations on Al Jazeera's Frost Over The World on Saturday.
"We are negotiating ... I had to campaign for an end to civilian casualties because we are a sovereign country and the Afghan people expect their government to stand for them," Karzai said.
Barack Obama, the US president, is expected to approve the deployment of about 30,000 extra US troops to Afghanistan in the next few days. There is currently 80,000 US and Nato soldiers in the country, backed up by the Afghan army and police.