While Kosovo maintains it will declare independence, Serbia says it will "annul" such a move.
Frank Wisner, the US envoy at the talks, urged both sides to keep their pledge to preserve peace and dialogue beyond the end of formal negotiations.
Ischinger said on Monday there would be "no possibility for further negotiations".
The three-day meeting between Serbs and the Albanians was the sixth and last since late August.
After eight years under UN control and Nato protection, and with no compromise in sight, the US and the EU believe independence under EU supervision is the only viable solution to the dispute.
The mediators from the EU, the US and Russia will make final visits to Serbia and Kosovo next Monday before they submit their report to the UN on December 10.
"Unfortunately no deal was reached," Fatmir Sejdiu, Kosovo's ethnic Albanian president, said.
He said the province would go ahead with a plan to declare independence with Western backing.
Hashim Thaci, Kosovo's prime minister-in-waiting, expressed regret that no agreement was possible with Serbia.
Slobodan Samardzic, Serbia's minister for Kosovo affairs, told Al Jazeera that any declaration of independence would violate both constitutional and international law.
"We shall proclaim it as illegitimate," he said. "We shall act as we did till now, we shall develop the Serbian institution within Kosovo with Serbs which are loyal to Serbia as a state."
Boris Tadic, the Serbian president, insisted compromise was still possible, and warned the ethnic Albanian majority against a unilateral declaration of independence within months.
Tadic argued that such a declaration of independence would have a "domino effect" across the region and beyond.
"For Serbia this doesn't mean that compromise is impossible," he said.
Vojislav Kostunica, the Serbian prime minister, said the independence demanded by Kosovo's 90 per cent ethnic Albanian majority cannot be legally obtained by a unilateral declaration backed by the West.
Serbia's ally Russia has already blocked independence in the security council and says it will "insist" on further negotiations.
The US and the EU say the troika's mandate ends once the report has been submitted.