Hang Soth, , who heads the agency that manages the Preah Vihear temple, said: "At first about 20 troops entered a pagoda in Cambodian territory. Later they increased their numbers to about 40."
But Seni Chittakasem, a local governor who said he had led a delegation into Cambodia to secure the release of the three protesters, insisted that the soldiers had remained nearby but on Thai territory.
He said that the three Thai protester, which included one Buddhist monk, had been freed.
"All three detainees have been released and now are on the Thai side," he said.
The protesters had placed wooden planks over barbed wire on the border to get across, vowing to reclaim the temple, which the World Court handed over to Cambodia in a 1962 ruling.
The incident comes shortly after Thailand's foreign minister resigned over the temple row.
Noppadon Pattama was the second cabinet member to resign in two days, increasing pressure on the government of Samak Sundaravej, the Thai prime minister, who was installed just five months ago.
Democratic Party politicians had accused Noppadon of supporting Cambodia on the issue of Preah Vihear, a temple located on land that both countries claim as their own.
Unesco, the UN cultural organisation, granted World Heritage status to the temple last week.
But on the same day Thailand's constitutional court ruled that Noppadon acted unconstitutionally when he endorsed Cambodia's Unesco application without first consulting the Thai parliament on the matter.
Critics said the endorsement undermined Thailand's own claim to land near the temple.
The 11th-century Preah Vihear has been at the centre of a long-running territorial dispute as the main compound lies inside Cambodia but its most accessible entrance is at the foot of a mountain in Thailand.