Ashdown accepted the job in mid-January after Ban Ki-moon, the UN secretary-general, asked him to be overall co-ordinator of international aid, government and political efforts in Afghanistan.
Foreign ministers representing Nato countries had envisaged combining the roles of UN, Nato and European Union representatives into one single role.
Ashdown, who was Bosnia-Herzegovina's postwar international administrator, said he initially had the support of Karzai.
"I wouldn't have dreamed of undertaking the job unless (Karzai) agreed that I should do so," he said.
"Now something's happened that's changed his view. I think that's far more to do with Afghan internal politics than it has with the international community."
Asdown has said the idea of a "super envoy" has effectively been dropped due to the difficulties of integrating UN, EU and Nato decision-making processes.
Media reports say Karzai changed his mind in the past few days and raised objections with Gordon Brown, the British prime minister, and Condoleezza Rice, the US secretary of state, during meetings at the World Economic Forum in Davos, Switzerland.
Brown's office declined to comment on the private meeting.
An EU diplomat also said Karzai had conveyed reservations about Ashdown to Brussels last week.
But Rangeen Dadfar Spanta, Afghanistan's foreign minister, said on Sunday that Kabul's objections were not down to Ashdown or his nationality but to a "negative atmosphere" created around the envoy role.
For Ashdown, the Afghan role would have been more limited to the sweeping powers Ashdown held in Bosnia.
As a top international administrator in Bosnia from 2002 to 2005, he was responsible for enforcing a peace accord and had the power to fire Bosnian officials as high as presidents and to impose and annul laws.
Zahir Tanin, Afghanistan's ambassador to the UN, said Afghanistan's preferred choice for the role now was British General John McColl, Nato's deputy supreme commander in Europe.
McColl commanded the first international security force in Afghanistan in 2002.