"The establishment of effective governance is a critical enabler for improving development and security."
Part of the problem is the widespread perception among Afghans of Karzai's government being corrupt and inefficient.
"While Afghanistan has achieved some progress on anti-corruption, in particular with regard to legal and institutional reforms, real change remains elusive, and political will, in particular, remains doubtful," the report said.
The killing of Afghan civilians by US and other foreign forces has also been a polarising issue.
In the latest incident, hundreds of people protested on the streets of the city of Jalalabad, eastern Afghanistan, on Thursday over the death of a relative of an Afghan member of parliament.
That popular anger, combined with a spike in violence during the presidential election last August, has allowed the Taliban to "perceive 2009 as their most successful year," the US report said.
"Expanded violence is viewed as an insurgent victory, and insurgents perceive low voter turnout and reports of fraud during the past presidential election as further signs of their success," it said.
According to Pentagon figures, violence is "sharply above the seasonal average for the previous year - an 87 per cent increase from February 2009 to March 2010".
Despite the rise in violence though, the report also cited opinion polls showing that more Afghans were feeling safe, with 84 per cent saying security levels were "fair" or "good".
A US defence official briefing journalists on the report said it showed that "after a number of years of things moving in the wrong direction ... we are no longer moving in the wrong direction and there are signs we are moving in the right direction".
Pentagon said with an increase in US forces in the country, Taliban fighters were now coming under "unprecedented pressure".
"This strain has been compounded by the recent high-profile arrests of several Pakistan-based insurgent leaders by Pakistani authorities and removal of many Afghanistan-based commanders," it said.
However, the US defence official added that there was no indication of any leadership crisis within the Taliban hierarchy.