Byers said: "We are still extremely concerned about overcrowding in the camps and especially in Manik Farm camp.
"The situation in this camp is improving slowly ... but conditions in these camps remain extremely critical, extremely difficult, which is why we are asking for total access."
Ban is due to visit the Manik Farm complex in the northern district of Vavuniya, where most of the people displaced by the recent fighting are housed.
The government describes the camps as "welfare villages", but Tamil activists have likened them to concentration camps.
The Sri Lankan government imposed restrictions on access to the site last week, shortly before it declared victory over the separatist Liberation Tigers of Tamil Eelam (LTTE) after more than 25 years of fighting.
The government has said that the displaced Tamil civilians, many of whom left the conflict zone in a malnourishered state, will be rehoused within six months.
"The government of Sri Lanka ... outlined a 180-day plan to resettle the bulk of IDPs [internally displaced persons] to their original places of habitation," it said in a joint statement with Indian officials.
There have been accusations of child abductions inside the camps, with the Coalition to Stop the Use of Child Soldiers saying it had received reports of pro-government paramilitaries taking those it said had fought for the Tamil Tigers.
Nimmi Gowrinathan, the director of South Asia Programmes for Operation USA in New York, an international relief organisation, told Al Jazeera: "If past visits by UN officials have been any sort of record, I don't think [Ban] is going to achieve what we need him to achieve at this stage for the humanitarian crisis.
"The Tamil people are hesitant to believe there is the political will to provide aid services."
Vijay Nambiar, Ban's chief of staff, has urged Sri Lanka to begin a process of reconciliation with the Tamil minority, which make up 12.6 per cent of the population of 20 million.
"The process of national reconciliation, we feel, must be all-inclusive so that it can fully address the legitimate aspirations of the Tamils as well as other minorities," Nambiar said.
"It is important that victory becomes a victory for all Sri Lankans."
The Tamil Tigers had been fighting for an independent homeland for ethnic Tamil sin the north and east of the island after years of marginalisation by the Sinhalese majority.
The UN estimates that up to 100,000 people have died during the conflict, including at least 7,000 civilians killed since the beginning of the year.
"[I lived] near Pudukuripu, in a place called Padapalaayam ... Every hour the Sri Lankan army fired 400 to 500 shells. We were forced to live inside the bunkers.
"My elder brother was injured in the shelling and he was taken to the hospital ... To remain alive and avoid all the atrocities I escaped to India.
"I escaped to India by boat, travelling with many unknown people."
Pradeepan, Tamil refugee
The government has not announced the civilian toll, but on Friday the government revealed that more than 6,200 soldiers had been killed and nearly 30,000 wounded in the last three years of fighting .
Gotabaya Rajapaksa, the defence secretary, announced the figures - which cover the phase of fighting against the LTTE since August 2006 - on state television.
"We made huge sacrifices for this victory," he said on Thursday.
The government has said that at least 15,000 LTTE fighters died in the conflict, but these figures are yet to be confirmed.
Residents of the capital, Colombo, are expected to march to parliament on Friday in support of the government's victory.