Hoda Abdel Hamid, Al Jazeera's correspondent in Islamabad, said the withdrawal followed talks between the Pakistan government and the Taliban.
"There was a meeting between Sufi Muhammad [the cleric who brokered a deal between the Taliban and the Pakistani government], some members of the Taliban and a few officials from the northwest frontier," she said.
"Tension was escalating and people were becoming worried about what a Taliban settlement in Buner would mean."
The Taliban's pullout decision came as General Ashfaq Kayani, Pakistan's army chief, sought to dispel doubts that the country's armed forces were reluctant to take on the fighters.
The army "will not allow the militants to dictate terms to the government or impose their way of life on the civil society of Pakistan", he said in a statement after a meeting of high-level military commanders.
Asif Ali Zardari, the Pakistani president, ratified a deal in April to put Malakand - home to about three million people in northwest Pakistan, including the district of Buner - under sharia, or Islamic law, as part of efforts to end a Taliban revolt.
Critics who attacked the Swat deal on the grounds that government "capitulation" would only embolden the Taliban, have said that the fighters' entry into Buner vindicates their fears.
Soon after hundreds of armed Taliban fighters set up checkpoints and occupied mosques in Buner, the US said it was "extremely concerned".
"I think the news over the past several days is very disturbing, the administration is extremely concerned," Robert Gibbs, a White House spokesman, said, noting what he called "candid" comments on the issue by Hillary Clinton, the US secretary of state.
|A barber stands by a 'Do not shave' warning placed by the Pakistani Taliban [AFP]
Some fighters in Buner reportedly moved into another nearby district, Shangla, on Thursday.
The Taliban's presence in Buner prompted emergency talks by political officials in northwest Pakistan, on what constitutes a "violation" of the Swat peace deal.
Mian Iftikhar Hussain, Pakistan's provincial information minister, said: "An all-parties conference convened by North West Frontier Province chief minister Ameer Haider Hoti has begun in Peshawar to discuss developments in Swat and Buner."
Those who attended the meeting included representatives of Hoti's Pashtun secular nationalist Awami National Party (ANP), which belongs to the federal government coalition, and Pakistan's two main Islamist parties, Jamat-i-Islami and Jamiatul Ulema-i-Islam.
The Pakistani Taliban, under the leadership of Maulana Fazlullah, launched a campaign nearly two years ago to enforce sharia in the Swat valley, beheading government officials, destroying girls' schools and forcing thousands of people to flee.