The closed-door talks lasted nearly eight hours.
No troops increase
It was decided that troops on the border would not be increased and that negotiations would continue at an unspecified time.
Thailand has more than 500 and Cambodia about 1,000 troops on the border.
The 11th-century temple was built during the reign of King Suryvarman I during the 600-year Khmer empire.
Built to honour the Hindu god Shiva, the temple has withstood decades of war.
In 1998, hundreds of Khmer Rouge guerrillas made their final surrender at the temple.
Unesco deemed it a World Heritage site for its location, rare architecture, religious function and carved stone ornaments.
General Boonsrang Niumpradit, the Thai supreme commander, said that both nations' troops would "not use any violence or weapons".
"We both have legal problems, which we have informed our superiors to discuss."
Cambodia maintains that Thai troops are trespassing on its territory.
Last week, hundreds of Cambodian and Thai troops massed at the border and came close to a shootout as the confrontation escalated over the territory near the temple complex.
Hun Sen, the Cambodian prime minister, said on Saturday in a letter to Samak Sundaravej, his Thai counterpart, that a previous international court ruling showed that Thailand did not have a claim to the territory.
The letter said the map used in the 1962 ruling by the International Court of Justice shows that the temple "is legally located approximately 700m inside Cambodian territory".
"Nonetheless, I have full confidence that our joint efforts will result in a mutually satisfactory solution to [our] current problem," Hun Sen said.
The conflict over the territory surrounding the ancient Hindu temple escalated earlier this month when Unesco, the UN cultural body, approved Cambodia's application to have the complex named a World Heritage site.
Thai activists say the status undermines Thailand's claim to the compound of a nearby Buddhist pagoda.