He said that 12 fast-attack craft went to inspect the boats when the Tigers opened fire on them.
Nanayakkara told reporters: "In total, 11 Tiger boats were destroyed. We can't say exactly, but we estimate around 40 Tigers were killed."
The military has given no details of any navy casualties from the clash and there were no independent accounts of what had happened. The Tigers were not immediately available for comment.
The clash was the latest in a series of land and sea battles.
The military has reported killing hundreds of Tigers in recent weeks, with the death toll from renewed fighting well over 5,000 since early last year.
But, analysts say both sides often exaggerate enemy losses and play down their own.
The government of Mahinda Rajapaksa, the Sri Lankan president, has vowed to destroy the Tigers' military assets and to clear the group from territory it controls in the island's north. The army captured the Tigers' eastern strongholds earlier this year.
Military analysts say there is no end in sight to the conflict. At least 70,000 people have died since 1983 in the separatist struggle.