A statement by the South Korean Joint Chiefs of Staff said on Monday three North Korean patrol boats crossed the so-called Northern Limit Line in two separate incidents in the space of just over an hour.
Shots were fired three times in an incident involving two North Korean vessels, the statement said.
The North Korean boats did not return fire.
"Three North Korean patrol boats crossed the Northern Limit Line at two locations and retreated north after warning messages and shots from our navy," the statement said.
It said there were about 80 Chinese fishing boats in the area at the time.
The third North Korean boat retreated after a radio warning, it said. The incidents took place between 11am (0200 GMT) and noon.
Dealers said the Won currency dipped slightly against the US dollar on the news, although the stock market was unmoved.
The latest incursions follow a series of incidents along the border between the two Koreas - the world's last Cold war-style frontier - and around the peninsula's coast.
The US cut its troops in a deal
for Seoul to guard its own border
Last week, South Korean guards found three holes in fences in the Demilitarised Zone.
Two fishing boats strayed south before that - one on each coast - and the South also hunted in vain for two North Korean submarines off the south of the peninsula.
Diplomatic sources say North Korea could be carrying out training for special forces or testing Washington and South Korea's mettle ahead of Tuesday's US presidential election.
The Northern Limit Line was drawn by US-led United Nations forces after the 1953 Korean war armistice. Pyongyang has demanded a new sea frontier be drawn far to the south, and in 1999 declared the line invalid.
At midnight on Sunday, the United States relinquished its last outpost in the Demilitarised Zone to South Korea, and cut its troops there as part of a deal to give Seoul more responsibility for guarding the border with North Korea.
US troop presence
The US sharply cut its troop presence in the Panmunjom truce village, also known as the Joint Security Area, to fewer than 40 from 220, although the US-led UN force will still command the 550-strong unit.
"As previously agreed, transferring the role of security work at Panmunjom has been completed," a spokesman for the US military command told Yonhap news agency.
"As previously agreed, transferring the role of security work at Panmunjom has been completed"
Spokesman for the US military command in South Korea
The US military also handed over nearby Observation Post Ouellette, its last remaining guard post along the border and a popular destination for US presidents wanting a close-up glimpse of North Korea just metres away.