No casualties were reported in the second similar incident this month, AFP reported. An unidentified South Korean military spokesman said the ship was not military.
"We fired five shots at 02:53 GMT and the North Korean ship returned at 02:58 GMT," the statement said.
This was the 15th incursion by a North Korean vessel across the so called Northern Limit Line (NLL) boundary. It was the fourth time South Korea has fired shots to repel the boats.
Tensions on the Korean peninsula were last year heightened after the US government said the North admitted pursuing a secret nuclear arms programme in contravention of the nuclear non-proliferation pact, of which it was a signatory.
They have been further aggravated by the announcement Monday by Pyongyang that it will not dismantle its nuclear weapons unless United States abandons its “hostile policy” towards the country.
The official KCNA news agency demanded on Monday that Washington sign a non-aggression pact with North Korea, establish diplomatic ties and make clear it would not hinder Pyongyang’s foreign trade.
"If the US does not express its will to make a switchover in its policy towards the DPRK, the DPRK will have no option but to declare that it cannot dismantle its nuclear deterrent force at the talks," KCNA said.
Talks over the North’s nuclear programme are due to begin at the end of this month in China. The delegates will include representatives of the two Koreas, the US, China, Japan and Russia.
"China stands ready to promote the process of dialogue so as to ensure peace and stability of the Korean peninsula"
Any hopes of mediation were dashed last year when North Korea demanded bi-lateral discussions with the US.
The US responded, saying it would only participate if other countries in the region were present at the negotiating table.
Still, last month North Korea agreed to a multi-lateral convention.
"As the host, China stands ready to make all the preparations and work with the other parties to promote the process of dialogue so as to ensure peace and stability of the Korean peninsula," China said in a statement earlier this month.
In a separate move, and one that is bound to increase pressure on North Korea, Washington is planning a joint naval exercise with South Korea.
The New York Times said the focus of the training would be intercepting arms and other materials.
The naval exercise, which will occur in the Coral Sea northeast of Australia, is not officially directed at any one country, but is intended in part to put pressure on North Korea.
“We are not saying which countries are being targeted, because it would not be politically wise,” an Asian diplomat said in reference to the planned naval exercise, according to The New York Times.
“But the American government believes that one of the reasons why North Korea has agreed to the six-party talks in Beijing is that they are feeling the pinch.”
Part of the exercise will involve the “non-permissive boarding” of ships suspected of holding drugs, missiles and nuclear materials.