The Philippine government and the Moro Islamic Liberation Front (MILF), Philippines' largest armed group, have failed to reach an agreement on the creation an ancestral home for three million Muslims in the country's south.
The MILF considers such an agreement to be vital for the resumption of formal peace talks.
At least 120,000 people have been and two million displaced during the group's 40-year conflict with government forces on the island of Mindanao.
Mohaqher Iqbal, the MILF chief negotiator, said on Friday: "We failed to settle the old issues after two days of hard bargaining".
Iqbal's statement comes after talks in Kuala Lumpur were brokered by the Malaysian government.
"The talks collapsed because the government was undoing already-settled issues. The signing ceremony set for August 5 was cancelled," Iqbal said.
"They're trying to re-open discussions on what had been agreed upon."
The breakdown came a week after both sides reported making a breakthrough on the issue following several days of talks.
However, Rodolfo Garcia, the head of the government's peace panel, said that the talks were not dead.
"We had some disagreements when we were going over the final draft of the agreement on ancestral domain," he said.
"But I don't think there is a complete breakdown of talks. I'm still hopeful that we could save the negotiations."
Malaysian and Philippine foreign ministers had been due to witness the signing of the pact on August 5.
A Malaysian government official said the two sides became deadlocked over the issue of territorial rights.
"To everyone's surprise, the Philippine government re-visited the territorial issues which took us 14 months to resolve," the official said.
"The territorial issue ... has created an impasse and led to the collapse."
Manila and the 11,000-member MILF have been talking for more than a decade on how to give Muslims in the south more self-rule. It took nearly four years just to negotiate the expansion of an existing autonomous region.
However, after the pact's collapse, no further talks have been scheduled and both sides will return home on Saturday.
Gloria Macapagal Arroyo, the Philippine president, has repeatedly said she wants peace, but some in her cabinet are opposed to giving large swathes of land to Muslims, and politically powerful Christian groups in the south would oppose a final deal.
In Manila, pro-government legislators moved a bill in the House of Representatives on Thursday to postpone August 11 elections in the current Autonomous Region in Muslim Mindanao (ARMM).
The five legislators said elections to the ARMM should henceforth be synchronised to the three-yearly congressional elections which are held in the second week of May.
The next congressional election is due in May 2010.