Clashes began on Sunday night when the navy spotted 25 Tamil Tiger ships sailing south, Navy Commander D K P Dassanayake, a Sri Lankan army spokesman, told AP news agency.
Up to 70 fighters, in boats loaded with arms and ammunition, were killed, while one navy vessel was damaged and five sailors were injured in the attack, he said.
Several more boats retreated following the battle, just off the coast of the eastern town of Pulmoddai, about 225 km from Colombo, the Sri Lankan capital.
Percy Perera, chief inspector of police for Sri Lanka's Centre for National Security, said that 11 boats were sunk in the incident.
He also said that the navy believed a top Tiger naval commander was killed or injured during the clash.
Tamil Tiger fighters have not issued a response to the Sri Lankan army's statement.
More than 100 Tigers were killed in two separate sea battles earlier this month as they reportedly tried to reinforce positions in the east after losing territory to government forces.
Foreign mediators are attempting to hold together an increasingly fragile four-year old Norwegian-brokered ceasefire, which has unravelled following clashes that have killed at least 1,000 combatants and more than 100 civilians since July.
Earlier this month Norway said that the two sides had agreed to meet for talks for the first time since the Tigers pulled out of negotiations in April.
The Tamil Tigers began fighting in 1983 for a separate homeland in the north and east for Sri Lanka's largest ethnic minority.