It was hoped that the three-day talks, which ended on Thursday, would resolve nearly 40 years of conflict.
There was progress, however, on the sharing of natural resources and governance in the Muslim homeland on the southern island of Mindanao.
Government negotiators and the Moro Islamic Liberation Front (MILF) said in a statement: "It achieved significant progress ... with the exception of territorial delimitation, which was highly technical in nature and requires further deliberation."
Further thought needs to be given to verifying data on the ground, the parties said.
They did not say when negotiations, hosted by Malaysia, would resume. Both sides had hoped to break the deadlock this week and fix a date for the signing of an agreement on ancestral domain, which would pave the way for a final peace pact by September.
Thursday's statement gave no indication if the failure to resolve the territory issue would delay the pact beyond September.
The MILF has been fighting for an independent state for Muslims on Mindanao, although a truce has held since July 2003.
Negotiators from both sides had in February agreed to a preliminary deal on ancestral domain, the key to ending the revolt that has cost more than 120,000 lives.
The 40-year-old Muslim rebellion
has cost 120,000 lives
The revolt has stunted development of resource-rich Mindanao and clouded the overall investment and security climate of the Philippines.
Ancestral domain involves issues such as territories in the south that would be part of the Muslim ancestral homeland, the sharing of revenues from strategic resources such as gold, copper and oil and the form of government within the area.