The ICRC said it managed to ferry out 240 sick and injured people from Puttumatalan.
Exclusive footage broadcast by Al Jazeera on Thursday showed people attempting to flee the island with the help of the aid organisation.
The ICRC said the deaths of 16 patients by artillery fire three days ago highlighted the dangers faced by civilians in the war zone.
Efforts are now under way to ensure the safe passage from the island of about 160 people, officials said.
Sister Mary Colostica, a 74-year-old Catholic nun, said she was shot at as she tried to shepherd more than 2,000 civilians from village to village.
"When we tried to leave, the LTTE didn't allow civilians to leave and said only we [the nuns] can leave ... so we stayed back with the civilians," she said from a hospital in Trincomalee, the eastern port where she and the others were taken by the ICRC.
The civilians were treated at a makeshift field hospital before moving towards Puttumatalan to escape the fighting taking place nearby.
Humanitarian agencies say about 250,000 people are trapped between Sri Lankan government forces and the Tamil Tigers.
Colombo says the number is half of that but it is preparing to house 200,000 refugees at five "welfare villages" for up to three years.
The LTTE has denied accusations that it has kept civilians inside the war zone to use as fighters.
The group has lead a 25-year war against Colombo in an attempt to secure greater rights for ethnic Tamils on the majority Sinhalese island.