"It's too risky to stay at Government House because of repeated attacks against us," Anchalee Paireerak, a PAD spokesman, said.
"All of us have started to move now. We expect to complete the movement this evening."
An AFP reporter saw dozens of supporters carrying plastic bags and sleeping mats leaving the compound, which protesters had heavily fortified with razor wire and tyres.
In an apparent climbdown, the PAD has allowed 37 empty aircraft to fly out of Bangkok's Suvarnabhumi international airport in the past two days.
Al Jazeera's Step Vaessen, reporting from Thailand at a military base serving thousands of stranded travellers, said there were "extraordinary scenes", adding that "if you look at all those people trying to leave [the country], it looks like a natural disaster has struck Thailand".
More than 160,000 people have been stranded and the military base does not have enough check-in counters and the capacity to deal with all the luggage, our correspondent said.
"There have been fights inside the base because people are getting really angry," she said.
The PAD seized the cabinet offices in late August, as part of a campaign they launched in May to topple an elected government they accuse of running the country on behalf of Thaksin Shinawatra, a former prime minister ousted in a 2006 coup.
They took that campaign to unexpected heights last week, storming Suvarnabhumi airport on Tuesday and then occupying the older Don Mueang domestic airport the next day.
The abandonment of the premier's offices is likely to ease the risk of clashes with a rival pro-government group that camped out in Bangkok for a second day a few kilometres from Government House.
About 15,000 pro-government activists wearing red shirts had converged on city hall in downtown Bangkok on Sunday night, and around 1,500 of them remained there on Monday morning, police said.
"We will rally again this afternoon at the same place," Chinawat Haboonpard, a leader of the pro-government group, told the AFP news agency.
Chinawat said the group had not yet decided whether to launch a blockade of the Constitutional Court, which is due Tuesday to wrap up a case that could see the ruling party disbanded for vote fraud and Somchai banned from politics.
Officials said that once the protest ends, it could take a week to get the main airport up and running again because security and computer systems had been compromised during the blockade.
"Normally, checking the IT systems takes one week," Serirat Prasutanond told the Reuters news agency.
It will take longer if there were repairs to the airports massive computer systems, he added.
"We have to check, recheck, check, recheck. They want anarchy so that the army is forced to intervene and stage a coup."
|Police have been reluctant to use force on the protesters to end the siege [EPA]
Meanwhile, the Federation of Thai Industries has estimated the airports takeover is costing the country $57m to $85m a day.
Andrew Stotz, a security analyst, told Al Jazeera that the Thai economy will be hit hard.
"The current situation will hit tourism, business investments and those trying to export and import goods," he said.
"Thailand has had an economy that's been on the decline for a long time. While the country has had many political crises, we are at the peak of this current quagmire."
The airport siege, coming in the wake of a global economic slump, has also hurt Thailand's $15bn tourism industry.
The government estimates that the number of visitors could be cut by half if the crisis drags on, leaving about one million Thais jobless.
But PAD protesters see that as a price worth paying.
"Obviously it hurts the economy, but it's the only way we can push out this government. We have to sacrifice something," Prathan Tandavanitj, 60, told the Reuters news agency at the airport.