The battle erupted after a Tamil Tigers sank a navy patrol boat off the northern coast on Thursday as it escorted a troop transport carrying more than 700 soldiers.
The navy retaliated by sinking five rebel vessels, while the air force launched strikes on guerilla positions.
The Sri Lankan air force scrambled jets and helicopter gunships to the area in response to the worst outbreak of violence since a ceasefire in 2002.
At least 50 Tiger guerrillas were on the sunken rebel boats and all were believed dead, navy spokesman DKP Dassanayake said.
A search was going on for the missing sailors who "made worthy efforts to save hundreds of soldiers who were on board the main vessel," Dassanayake said.
There are now fears that the country could lapse back into civil war that raged for 20 years prior to the ceasefire.
Daya Master, a media co-ordinator for the rebels, said: "The air force has bombed our territory, but nothing has fallen here. We have no word on casualties yet."
"This is a very serious attack (by the Tigers), a blatant violation of the cease-fire agreement," government spokesman Keheliya Rambukwella told AP.
'Fringes' of war
The naval battle was about 50km from the Liberation Tigers of Tamil Eelam's northern stronghold.
Helen Olafsdottir, spokeswoman for the unarmed Sri Lanka Monitoring Mission, said: "Our monitors saw several Tiger boats attacking the transporter ship and firing.
"We have a monitor on that boat."
"Our monitors saw several Tiger boats attacking the transporter ship and firing. We have a monitor on that boat"
Sri Lanka Monitoring Mission spokeswoman
The transporter sustained light damage and moved to shelter in Indian waters after the confrontation.The Tigers, who are fighting for a separate homeland for ethnic Tamils in the north and east, have withdrawn from peace talks indefinitely, and said on Tuesday that Sri Lanka was moving towards the "fringes" of war.
In a separate incident in the eastern district of Trincomalee, the military said rebels had fired mortars and bursts of gunfire at a navy post, but that no one had been hurt.
The violence comes after a rash of attacks last month and a day after a Japanese peace envoy left the island after an failed attempt to persuade the Tigers to return to peace talks.