About 265,000 ethnic Tamils were displaced in the military's recent offensive against the Liberation Tigers of Tamil Eelam, or LTTE, and many of them have been sent to overcrowded camps in the country's north.
"Since last weekend there have been additional restrictions imposed on aid organisations, including ICRC," Paul Castella, the head of the group's Sri Lanka operations, said.
"The authorities have said that because of security they had to restrict access to certain areas.
"What is the take of these civilians and what the conditions are we don't know because we are not granted access to the area."
Separately, rights groups have called on Sri Lanka to release three Tamil doctors detained after government forces overran the LTTE's last outpost in the country's northeast.
During the final stages of the conflict, journalists were prevented from entering the war zone and the international media came to rely on the reports of Thurairaja Varatharajah, Thangamuttu Sathyamurthi and V Shanmugarajah, three Tamil doctors working in the war zone.
The men were arrested on accusations that they gave false information about the number of casualties to the media. Police said that they can be held for up to 90 days.
Yolanda Foster, a Sri Lanka expert at the human rights organisation Amnesty International, told Al Jazeera: "Two of the doctors are now being held in the terrorist investigation division in Colombo but no formal detention order has yet been issued.
"That's a key concern because without a formal order, the relatives of the doctors do not know where they are.
"If they are not to be charged formally then they should be released."
The third doctor is believed to be detained in hospital after being injured while escaping fighting on Friday. However, this has not yet been confirmed.
Frank Donaghue, who heads Physicians for Human Rights, said: "We call on the Sri Lankan government to release the doctors immediately, and to respect their rights to legal counsel and to receive medical care as well as family visits."
"The Committee to Protect Journalists has also called for the doctors to be released.
"This is a chilling example of the intentions of the Sri Lanka government as it pursues its all out military solution in dealing with the [Tigers]," Bob Dietz, the group's Asia programme co-ordinator, said.
Mahinda Samarasinghe, Sri Lanka's minister for disaster managment and human rights, confirmed that the doctors had been "handed over" to the police.
"The Criminal Investigation Department has issued detention orders for the three doctors and investigations have commenced about possible collaboration with the LTTE," he told Al Jazeera.
Mads Gilbert, a doctor at the Norwegian Aid Committee who has worked in conflict zones, told Al Jazeera that the doctors had fulfilled their humanitarian obligations and that it was "outrageous" that they had not yet been given full protection they are entitled to under international law.
"They should immediately be handed over to the UN or ICRC and be protected from any interrogation or punishment from the Sri Lankan government."
He said that the doctors should be afforded witness protection since they are some of the few who are able to talk of alleged war crimes by the government.
"They are instrumental to know what is going on ... The most simple thing is for the government to execute them ... then they have silenced them completely," Gilbert said.
"These doctors are a threat to the government since they can tell the truth to the world."
President Rajapakse announced victory over the Tamil Tigers, who had fought for decades for a separate Tamil homeland, on Tuesday.
State television broadcast images of what it said was the body of Velupillai Prabhakaran, the LTTE chief, a bid to discredit claims by the Tamil Tigers that he was "alive and well".
Despite the government's declaration of victory, pockets of fighters are thought to still be active in Sri Lanka's east and officials suspect that sleeper cells are hidden in cities across the country.
Security forces were on high alert on Wednesday as the government declared a public holiday to mark the Tamil Tigers' defeat.
Meanwhile, concern has grown over the the high numbers of civilians killed and how the government will reach out to the country's Tamil minority.
Speaking in Geneva, Ban Ki-moon, the UN secretary-general, said that any serious allegations of war crimes "should be properly investigated".
"I remain concerned about the welfare and safety of the civilian population," he said.
The European Union has called for an independent inquiry into alleged human-rights violations carried out during the war.
The UN has said that between 80,000 to 100,000 people were killed in Sri Lanka's decades-long civil war.
Under international pressure to reach out to the Tamils, Mahinda Rajapakse, Sri Lanka's president, has vowed that a political solution to the island's ethnic divisions would be found.
|Rights groups have called for the release of three detained Tamil doctors [Tamil sources]
"All should live with equal rights. They should live without any fear or doubt," he said in his victory speech.
"Let us all be united."
He also called upon Sri Lankans, especially Tamils who fled the country, to return and help it rebuild.
"There are no minority communities in this country. There are only two communities, one that loves this country and another that does not," he said.
Ranil Wickremesinghe, the opposition leader, who has been criticised by the government for his conciliatory approach towards Tamils, called for national reconciliation.
"We have to have a discussion among ourselves, among the political leaders who represent the communities, and come up with a new Sri Lankan identity," he said from Brussels.