The incident lasted about 10 minutes before the Cambodians left, Cambodia's top army commander at Preah Vihear said.
"We exercised patience to prevent weapons from being fired," he said.
The standoff is the latest in a long-standing conflict over frontier territory that has never been fully demarcated.
Both countries have massed troops in the area despite the two sides agreeing to hold talks on Monday in the Thai capital aimed at soothing tensions and averting military action.
Reports said more than 800 Cambodian soldiers armed with assault rifles and rocket launchers and some 400 Thai soldiers had been deployed.
'Bad for relations'
Hun Sen, the Cambodian prime minister, wrote to his Thai counterpart, Samak Sundaravej, on Thursday saying that relations had been "worsening" since Thai troops "encroached on our territory" on Tuesday.
|Unesco declared the temple a World Heritage site last week [EPA]
"The deteriorating situation is very bad for the relations between our two countries," Hun Sen wrote, asking Samak to pull back his troops.
The confrontation comes after Unesco, the UN's cultural body, declared the 11th century Preah Vihear temple a World Heritage site last week.
In 1962, the International Court of Justice awarded the Preah Vihear ruins and the land it occupies to Cambodia, a decision that still rankles many Thais.
Thai activists and some government officials fear the temple's new status will jeopardise their country's claims to land adjacent to the site.
The border area of 4.6 square km has been at the centre of a long-running territorial dispute as the temple's main compound lies inside Cambodia but its most accessible entrance is at the foot of a mountain in Thailand.
The temple dispute has also led to heightened political tensions in both countries.
Cambodia is preparing for general elections on July 27 during which Hun Sen is expected to extend his grip on power after 10 years iin office.
In Thailand, there have been weeks of streets protests and severe criticism of the government led by Samak for supporting Cambodia's Unesco bid.