According to the South the North Korean boat then opened fire and the South's ship returned fire before the North's vessel sailed back toward its waters, apparently badly damaged.
South Korea's navy has said the ships were about 3.2km apart during the clash.
The naval clash occurred just hours before the Obama administration confirmed it was sending a senior envoy to North Korea for direct bilateral talks on the country's nuclear programme.
Stephen Bosworth will visit Pyongyang at a date yet to be announced but most likely by the end of the year, US officials said.
According to South Korean officials, the North Korean ship was on fire and heavily damaged following the clash.
Officials also said the South Korean ship was slightly damaged but suffered no casualties.
|South Korean defence officials say the North may take retaliatory measures [AFP]
South Korea's Chosun Ilbo newspaper citing unnamed sources reported that one North Korean officer was killed and three other sailors were wounded.
The Koreas blame each other for causing the skirmish in the rich crab-fishing area, where both sides regularly accuse the other of border violations.
North Korea's military said its ship was attacked by South Korean vessels as it was returning from checking on "an unidentified object that intruded" into its waters.
Chung Un-chan, the South Korean prime minister, said the North may have been trying to clamp down on Chinese fishing vessels in the area, and probably did not intend to violate the border.
However, some South Korean analysts say the North Korea was sending a message ahead of next week's tour of Asia by Barack Obama, the US president.
Robert Gibbs, the White House spokesman, told the Associated Press that the US administration was aware of the clash and urged North Korea not to escalate tensions.