The agencies are in negotiations with both the government and senior members of the Tamil Tiger rebels to reach civilians believed to be stranded there.
Peter Krakoling, deputy head of the delegation with ICRC said on Monday that "we are extremely concerned with the fighting in the town as we need a secure route so that we can go in and assess the situation."
Thousands more refugees arrived at the camps in Kantale on Monday after fleeing the ongoing fighting in Muttur, about 35km away, taking up to three days to walk.
Merrick Peiris, executive director of the Sri Lanka Red Cross Society, said "the number of refugees has doubled since yesterday in Kantale."
"Over the last twenty-four hours people have been arriving on foot. They have been walking for three days without water, women are carrying new born babies."
Peter Krakoling: The ICRC needs
access to assess the situation
Sanitation and basic facilities such as food and housing are being provided by national aid agencies and co-ordinated by the International Red Cross Society (IRCS).
The worry now is that there are more refugees headed for such camps, putting a strain on aid agencies.
Although relief efforts are able to provide basic living conditions in the short-term, the longer the fighting continues the more pressure the agencies will face.
"It depends how long the fighting continues, no one has a clear picture at this point," Stacey Winston, a spokesperson from the International Federation of the Red Cross said.
Line of fire
NGOs have not escaped the fighting after 15 aid workers from the French aid agency Action Against Hunger (ACF) were found dead on Sunday.
"Of course we are now a little nervous after the confirmed killing of fifteen aid workers from ….. We believe they were shot at point-blank range while working in their offices, this was confirmed by ACF," said Krakoling.
The Red Cross and other agencies
are overseeing relief supplies
"We are shocked that there are groups of people who will not shy away from taking steps to attack humanitarian workers," he said.
The aid workers are believed to have been local volunteers working on projects in tsunami-affected areas, but this has not been independently confirmed.
Ove Jansson, head of the Trincomalee district Sri Lankan Monitoring Mission, said "we haven't been able to get into Muttur because the roads have been mined and booby-trapped. No one has been able to get into Muttur to verify anything.
"But we do know that many have fled and the situation is very bad."
Two ambulances, which were donated by the Saudi Arabian Red Crescent were hit on Sunday.
Both the driver and patient were killed in one of the vehicles, which was completely destroyed.