However, pool reports from Pyongyang said the two did reach agreement on security arrangements for other cross-border reconciliation projects, but no specifics were given.
Among topics up for discussion were arrangements for the first regular cross-border rail service between the two Koreas.
The reports said both sides had agreed to discuss the issue of the fishing zone further in the future.
The sea frontier was demarcated by the American-led UN command at the end of the 1950-53 Korean War without consulting the North and Pyongyang has repeatedly demanded the border be redrawn further south.
News of the talks ending came as the chief US envoy to six-nation talks on North Korea's nuclear programme said the North had begun drafting a list of all its nuclear activities.
The process is the next key step toward total disarmament, Christopher Hill said after arriving in the South Korean capital, Seoul, on Thursday.
He said he would discuss the declaration with North Korean officials when he visits Pyongyang next week, and it would be reviewed at the next round of six-nation talks in Beijing early next month.
"We can have a verifiable solution by the end of the year"
Christopher Hill, chief US nuclear envoy
"I think it's pretty close to being ready", Hill said upon arriving in South Korea.
He said the list of nuclear programmes will have to be complete, saying it must include detail of the North's alleged work in helping Syria on a nuclear programme.
In late 2002 the US accused the North of seeking to secretly enrich uranium in violation of an earlier disarmament deal, allegations Pyongyang has denied.
Hill said North Korean officials however had promised during talks in August to address the issue to Washington's satisfaction.
Later the same day, Hill told American business leaders in Seoul that Washington has had "very detailed conversations' with Pyongyang on the uranium issue.
"While we do not yet have a solution as I stand here today, I'm confident that based on the direction of these conversations, we can have a verifiable solution by the end of the year," he said.
Hill said the aim was not "to humiliate anybody in this process" but to have an acknowledgment of the North's full uranium processing capability.