The main camp, known as a Manik Farm, holds a vast expanse of white tents neatly lined up in rows on dirt paths. Roads between the tents are crammed with people, and barbed wire fences encircle the area, keeping the tens of thousands of civilians from getting out. Soldiers are stationed across the camp.
Ban is to hold talks with Mahinda Rajapakse, the president, and Rohitha Bogollagama, the foreign minister, during his 24-hour visit to Sri Lanka.
Elisabeth Byrs, a spokeswoman for the UN humanitarian co-ordination office, said that overcrowding was the main problem in the camps, and in the Manik Farm camp in particular.
"The situation in this camp is improving slowly ... but conditions in these camps remain extremely critical, extremely difficult, which is why we are asking for total access," she said.
The government describes the camps as "welfare villages", but Tamil activists have likened them to concentration camps.
The Sri Lankan government imposed restrictions on access to the site last week, shortly before it declared victory over the LTTE after more than 25 years of fighting.
The government has said that the displaced Tamil civilians, many of whom left the conflict zone in a malnourishered state, will be rehoused within six months.
"The government of Sri Lanka ... outlined a 180-day plan to resettle the bulk of IDPs [internally displaced persons] to their original places of habitation," it said in a joint statement with Indian officials.
There have been accusations of child abductions inside the camps, with the Coalition to Stop the Use of Child Soldiers saying it had received reports of pro-government paramilitaries taking those it said had fought for the Tamil Tigers.
Nimmi Gowrinathan, the director of South Asia Programmes for Operation USA in New York, an international relief organisation, told Al Jazeera: "If past visits by UN officials have been any sort of record, I don't think [Ban] is going to achieve what we need him to achieve at this stage for the humanitarian crisis.
"The Tamil people are hesitant to believe there is the political will to provide aid services."
Vijay Nambiar, Ban's chief of staff, has urged Sri Lanka to begin a process of reconciliation with the Tamil minority, which make up 12.6 per cent of the population of 20 million.
"The process of national reconciliation, we feel, must be all-inclusive so that it can fully address the legitimate aspirations of the Tamils as well as other minorities," Nambiar said.
"It is important that victory becomes a victory for all Sri Lankans."
The Tamil Tigers had been fighting for an independent homeland for ethnic Tamils in the north and east of the island after years of marginalisation by the Sinhalese majority.
Earlier on Friday, thousands of people gathered in Colombo, to celebrate the victory.
Rajapaksa addressed the crowds, telling them that he was not afraid of calls for an investigation into possible war crimes during the conflict.
"[I lived] near Pudukuripu, in a place called Padapalaayam ... Every hour the Sri Lankan army fired 400 to 500 shells. We were forced to live inside the bunkers.
"My elder brother was injured in the shelling and he was taken to the hospital ... To remain alive and avoid all the atrocities I escaped to India.
"I escaped to India by boat, travelling with many unknown people."
Pradeepan, Tamil refugee
"They wanted to take [us] to international criminal courts. Some are trying to do this even now," he said.
"But I am not afraid of walking up to any gallows, having defeated the world's worst terrorists, and I know that I have the confidence and the strength of my people."
The UN estimates that up to 100,000 people have died during the conflict, including at least 7,000 civilians killed since the beginning of the year.
The government has not announced the civilian toll, but on Friday the government revealed that more than 6,200 soldiers had been killed and nearly 30,000 wounded in the last three years of fighting .
Gotabaya Rajapaksa, the defence secretary, announced the figures - which cover the phase of fighting against the LTTE since August 2006 - on state television.
"We made huge sacrifices for this victory," he said on Thursday.
The government has said that at least 15,000 LTTE fighters died in the conflict, but these figures are yet to be confirmed.