A navy spokesman said more than 40 Tigers were "presumed dead".
"We destroyed five boats and damaged seven to eight more.
"They were trying to attack our patrol boats, but they failed," DKP Dassanayake said, adding that no navy vessels were damaged.
The spokesman said 24 Tiger boats, including four suicide boats, were involved in the attack and the Sri Lankan navy was trying to recover one boat which was drifting.
The Liberation Tigers of Tamil Eelam, who are fighting for an independent state in the island's north and east, were not immediately available for comment and there was no independent confirmation of what happened or of any casualties.
Fighting is now focused on the north after the military captured the Tigers' eastern stronghold.
About 4,500 people have been killed since last year alone.
The government has vowed to destroy the Tigers militarily, while the separatists say they will step up attacks using a home-grown air force of light planes smuggled into the country in pieces.