The frontier was drawn by the United Nations at the end of the 1950-53 Korean War, and Seoul has repeatedly rejected the North's demands that it be changed.
South Korea's chief envoy, Major-General Jung Seung-jo told Kim it was "highly regrettable" that the negotiations at the truce village of Panmunjom would end without results.
"Your side continued making this demand even though your side knows very well that our side cannot accept it," Jung said, referring to the border issue.
The generals from the North and South left the room after the 40-minute session without shaking hands or setting any date for a next meeting – with Kim appearing visibly angry.
The South had called for the two sides to at least reach consensus on less controversial issues, such as opening a hotline between navy commanders to prevent future clashes in waters off the peninsula's western coast.
At the start of Thursday's meeting, Kim berated his South Korean counterparts, saying their attitude reminded him of an "Aesop's fable" about a liar.
"Throughout the past few days, I feel as if I had become a victim of April Fools' Day jokes," he said.
The talks are the highest-level dialogue channel between the two militaries, and were intended to follow up on agreements reached in May on setting up a joint fishing area around the contended border and co-operation on security arrangements for joint economic projects there.
Waters around the sea border are rich fishing areas and have been a past scene of deadly conflicts in 1999 and 2002.
The two Koreas have remained technically at war since 1950.