He told the assembled group the operation will be led by Afghan forces and will not resemble a heavy military offensive with tanks and airstrikes, which are blamed for inadvertently killing civilians.
"The cleaning-up operation will start first inside Kandahar city and then we will go to the districts," he said.
General McChrystal said that he was pleased that Karzai made such a "strong, clear call" for unity.
"He looked at the people and he asked them if they were willing to sacrifice and head the strength to do this and they came back with a strong resounding indication that they were and I think that that's critical as they go forward," McChrystal said.
Nato has already begun an operation to ramp up security in Kandahar, and the campaign is expected to accelerate the coming months.
But many of the estimated half-million Kandahar residents are sceptical of the operation, fearing it will lead to more bloodshed.
Al Jazeera's James Bays reporting from Kabul, said: "The Pashtun people who live in the south and east of Afghanistan and particularly the people around Kandahar are fiercely independent and very distrustful of the role of foreigners so there could still be some problems as the operation is implemented."