"[The town of] Daggar they say is now under control of military forces. That has prompted some people to go back. But the vast number of people who fled Buner are still holding back, [fearing] there could be another round of escalation."
The Pakistani military is fighting the Taliban in the Swat valley and Malakand along with Buner, and has vowed to expand its offensive to Waziristan.
The United Nations estimates that around 1.5 million people have fled their homes since fighting in Pakistan's North West Frontier Province (NWFP) started at the end of April, in addition to about 550,000 people already displaced by previous violence.
About 20 per cent of those displaced are living in government camps, while a vast majority of people are staying with relatives or in other private accommodation.
Al Jazeera's Mike Hanna, reporting from Swabi district near the capital, said many had moved in with "friends, family and complete strangers".
He said Pakistanis felt an obligation to help those who have fled, and that their willingness to help also was a sign of public support for the army.
"Those in the area who are lucky enough to have homes help those who are fleeing out of humanitarian compassion but also as part of a bolstering of popular support, as part of a duty for all Pakistanis who support their government in the struggle to retain a sense of national sovereignty.
"The efforts of the army continue to be supported. If those who are suffering away from their homes turn against the war, it is very damaging to Pakistan."
Yousuf Raza Gilani, the prime minister, asked local representatives of aid groups and other countries for additional financial aid at a donors conference on Thursday.
He said peace could not be achieved without proper help for the displaced.
Dominique Frankefort, emergency co-ordinator with the UN World Food Programme (WFP), said two million people would need food until at least September.
"There are very few non-governmental organisations and there is very little government assistance," he told the news agency AFP.
"We are catching up, if you have 200,000 additional IDPs [internally displaced persons] coming in per day you cannot feed them immediately."
The United States has pledged $110m in humanitarian aid to Pakistan.
An aircraft carrying air-conditioned tents and 120,000 pre-packed meals arrived on Wednesday, the US embassy said.
Some of the pledged money will also be spent on buying Pakistani wheat to boost the local economy.