Kenichiro Sasae, Japan's top negotiator, said he felt there was no prospect of an agreement late on Friday on the draft on how to implement a 2005 agreement that calls for Pyongyang to disarm in exchange for security guarantees and aid.
"There are some parts in which we had progress but on others we ran into difficulty," Sasae said after daylong meetings with the other delegates.
"We will continue with the talks, but at this point in time I don't feel there is a prospect of reaching an agreement."
Christopher Hill, the US assistant secretary of state, said: "I would say some of the tougher parts we got through fairly quickly."
He said that there were one or two issues of disagreement.
A diplomatic source close to the talks said the draft prepared by China stated that North Korea would "suspend, shut down and seal" nuclear facilities at the Yongbyon plant within about two months in return for energy and economic aid.
The fresh momentum in the talks between the two Koreas, China, the US, Japan and Russia came after the US and North Korean negotiators held path-breaking two-way talks in Berlin last month.
That meeting cooled tension that had boiled after Pyongyang staged its first nuclear test blast last October and the UN responded with sanctions.
North Korean view
Kim Kye-gwan, the chief North Korean delegate, said his team had been "able to reach agreement on some issues" with Washington.
"There are still differences on a series of issues in the overall talks, so we will try to work them out," he told reporters.
"You should not try to count the chickens before they hatch, as somebody said."
Pyongyang's tight-lipped diplomats have not publicly spelled out their price for accepting a shutdown of Yongbyon, which produces plutonium that can be refined for nuclear weapons.
But a Tokyo-based newspaper close to North Korea suggested that Pyongyang's rulers also want Washington to prove its goodwill.