D K P Dassanayake, a navy spokesman, said that the boat was spotted on Sunday morning, 220km off Sri Lanka's eastern coast.
The boat's crew refused to identify the vessel and then tried to escape, sparking an eight-hour gun battle in which the rebels allegedly hoisted their separatist flag.
The air force was called in to bomb the boat, and several navy battleships were also called for back-up.
Dassanayake said between12 to 15 suspected rebels were killed. The navy said there were no military casualties.
Navy intelligence indicated the boat was carrying artillery and missiles, he said.
Rebel officials could not immediately be reached for comment.
Tamil Tiger rebels meanwhile detonated a roadside bomb near Sri Lankan forces on a foot patrol on Sunday, wounding four, and shot to death an electricity board employee in an overnight attack, the military said.
The roadside explosion, on a major supply route in the eastern Trincomalee district, injured a soldier, a guard and two civilians. All four were being treated in a hospital.
An employee with the Celyon Electricity Board was killed and two others were wounded in the troubled northern Jaffna peninsula on Saturday night when a company vehicle came under rebel fire, the military said.