A spokesman for the Joint Chiefs of Staff said the North's land-based artillery batteries fired several rounds into waters near the border shortly after 9:00 am (0000 GMT), adding there were no immediate reports of damage or casualties.
"Our military immediately fired back in response," an unnamed official told the AFP news agency.
The Yonhap agency, citing an unidentified military official, said the North's artillery rounds landed north of the border.
The official refused to say what types of weapon were used or how many rounds were exchanged.
"Our military fired at the shells in the air," he said. "Our field manual states that we are supposed to target any incoming flying objects."
The North's shells landed near the South Korean-controlled island of Baengnyeong in the Yellow Sea, Yonhap reported, adding that South Korean marines based on the island responded by firing shoreline Vulcan cannons with a range of 3-4km.
"When the North fired, some 20-30 columns of water shot up into the sky," the agency quoted one source as saying.
The sea border has been a constant source of tension since it was drawn by United Nations forces at the end of the 1950-53 Korean War. The North insists it should be drawn further to the south.
On Tuesday South Korean officials said the North had declared a two-month ban on shipping in two zones around the border.
"North Korea will likely continue such low-intensity military provocations like this in the next few months"
Baek Seung-joo, Korea Institute for Defence Analyses
The announcement raised speculation that the North was preparing to hold military exercises or missile tests in the area.
Last month North Korea warned South Korean ships to avoid the border area, saying its coastal artillery would stage firing exercises in response to "reckless military provocations" from the South.
On Sunday the North's military lashed out at South Korea's vow to launch a pre-emptive strike to thwart any nuclear attack, calling it "an open declaration of war".
"North Korea will likely continue such low-intensity military provocations like this in the next few months," Baek Seung-Joo, of the Korea Institute for Defence Analyses, told AFP.
"But it is unlikely to take things to the extreme, as it in general wants to maintain economic co-operations with South Korea."
The two Koreas are technically still at war never having signed a peace treaty ending the 1950-53 Korean War.
South Korean markets on Wednesday regained ground after falling on news of the sea border incident, with analysts playing down any long-term impact.
The benchmark Kospi stock index pared most of its earlier one per cent fall and the won returned to positive territory following local media reports on the military engagement.
The Kospi rebounded to as high as 1,639.78 points after a session low of 1,620.55, compared with Tuesday's close of 1,637.34.
The won rose to as high as 1,158.0 per dollar from the previous close of 1,163.3.
Analysts said the news had a relatively big impact because investors were already jittery over China's monetary tightening and the debate on financial regulatory changes in the US.