Annan, while commending Sudan on Saturday for its achievements in Darfur, called for local authorities to exert more effort to improve the condition of hundreds of thousands of displaced people.
On a day-long visit to South Darfur state, he was briefed by Governor al-Haj Ata al-Mannan Idris on tribal reconciliation efforts aimed at restoring social cohesion and improving the lives of residents.
Annan conceded to the governor that the overall humanitarian situation was improving, but said more efforts were needed to restore Darfur to normality.
Later in the day, Annan toured a burned-out town and heard survivors describe how it was bombed in a government air raid.
Thousands of refugees welcomed the UN chief in Kalma Camp, the biggest in Darfur, before he travelled to Labado, also in South Darfur state, to wander among burned huts and speak to worried owners who have started returning home.
Donors pledged $300 million in
cash and more in kind for Darfur
"Now we are back but still we don't have security and we feel unsafe," Murra Ahmad told Annan after she described how five government planes had bombed Labado and driven her out.
The southern Darfur state had seen some of the worst recent violence in the three-year conflict.
In Khartoum on Friday, Annan met top Sudanese officials about Darfur.
"We discussed the need for us to do everything we can to bring security to Darfur, and ensure that the farmers can go back to their land, plant it, cultivate and harvest their crops. Otherwise we are going to have a major humanitarian effort which will stretch the capacities of the international community," Annan said after the meeting.
"We discussed the need for us to do everything we can to bring security to Darfur..."
Race against time
The secretary-general's three-day visit to Sudan immediately followed his attendance at an international donors' conference in Ethiopia, where Annan said the world was facing a race against time to prevent Darfur's situation from worsening.
By Friday, donors had pledged $300 million in cash and more in kind to help the African Union expand its peacekeeping mission in Darfur.
At least 180,000 people have died - many from hunger and disease - and about two million others have fled their homes in Darfur to escape the conflict, which erupted when rebels took up arms against what they say are years of state neglect and discrimination.
Sudan for its part has said that rebels have exploited an age old conflict between subsistence farmers and nomads over shortage of grazing land.