Dozens of rebels were also believed to have been killed in the fighting.
The clash at sea near the besieged army-held Jaffna peninsula comes amid a five-week bout of renewed civil war after four years of ceasefire, and as the army seeks to wrest control of rebel territory near a strategic port in the island's northeast.
A military spokesman said: "It was a major attack. There were 20 rebel boats. We were able to destroy 12 LTTE craft, including five LTTE suicide boats," a military spokesman said.
"They were humiliated in their so-called seas and withdrew."
The spokesman said he believed at least 75 Tigers had been killed, but there was no independent confirmation. The Liberation Tigers of Tamil Eelam (LTTE) was not immediately available for comment.
The military spokesman said two sailors were injured and two navy fast-attack boats were slightly damaged by gunfire in the battle, which raged through the night and into the early hours of Saturday.
He believed the Tiger flotilla had been seeking to attack a northern naval base at Kankasanturai (KKS) on the Jaffna peninsula, which is cut off from the rest of the island by rebel lines and where there are severe food shortages.
"I feel like they were doing something to disrupt KKS to damage supply lines to the north," he said.
The army is trying to take the Tiger-held town of Sampur, where the rebels are in artillery range of a major naval base in the northeastern harbour of Trincomalee and able to disrupt a key maritime supply route to Jaffna.
Fourteen soldiers have been killed and 92 wounded since that offensive began on Sunday. The army estimates about 120 rebels were killed.
The military said Jaffna itself was quiet after days of artillery battles, and residents - thousands of whom want to evacuate to Colombo after weeks being trapped in the enclave - were hopeful civilian flights would soon resume to the capital.
Reporters Without Borders meanwhile voiced concern at the abduction of another Tamil media worker just days after a Tamil journalist was kidnapped and later released. Several Tamil journalists have been killed so far this year.
The government is preparing to send a second shipment of food aid and emergency supplies to Jaffna, which is expected to sail on Sunday.
Humanitarian workers say the government is hampering the delivery of supplies such as medicines to rebel areas.
"We are being denied proper access to LTTE areas by the security forces and the government," one aid worker said.