"There were no casualties on our side while the North Korean boat, half-destroyed, sailed back to the North"
South Korean defence official
"A North Korean patrol ship crossed the Northern Limit Line and did not cease when we fired warning shots," South Korea's Yonhap news agency quoted a military source as saying.
The North Korean boat then fired back, the source said.
Yonhap said no South Korean casualties were reported in the clash, but officials said that the North Korean boat was badly damaged.
"There were no casualties on our side while the North Korean boat, half-destroyed, sailed back to the North," the South Korean military source told the agency.
The clash broke out at 11:28am (0228 GMT) near Daechong island.
Speaking to reporters in Seoul, South Korean Commodore Lee Ki-sik described the clash as "a regrettable incident", adding: "We are sternly protesting to North Korea and urging it to prevent the recurrence of similar incidents."
North Korea's military later issued a statement blaming South Korea for the clash, saying South Korean ships had crossed into North Korean territory, and demanding an apology.
The western sea border between the two Koreas was the scene of deadly naval clashes in 1999 and 2002 and has always been a potential flashpoint.
The clash in 2002 left six South Korean soldiers dead and others wounded.
The Northern Limit Line was drawn unilaterally by UN forces at the end of the Korean war in 1953, but the North has never recognised it and insists it should be drawn further to the south.
Aside from territorial sovereignty, the area is also a rich fishing ground for crabs.
North and South Korea have disagreed on the demarcation of their sea border for more than 50 years since the end of the Korean war.
That conflict ended in an armistice and not a permanent peace treaty and the two countries remain technically at war.
Last month, North Korea's navy accused South Korea of sending warships across the border to stir tensions, warning that what it called "reckless military provocations" risked triggering armed clashes.
The latest incident comes just days before a high-profile visit to Asia by Barack Obama, the US president, during which the ongoing tensions on the Korean peninsula and the North's nuclear programme are expected to feature high on the agenda.
US officials said on Monday that Obama had decided to send a special envoy to Pyongyang for rare direct talks on the nuclear weapons issue.
No date has been set for the visit but it would be the first one-on-one talks since Obama took office in January.