The group has been protesting across Bangkok, the capital, since May to demand the Samak's resignation.
The prime minister confronted the protesters, refusing their calls for his resignation.
"There is no way that a country of 63 million, like Thailand, will let a group of five people form a street gang to seize control," he said.
The PAD said the protests and the seizure of a state-run television station on Tuesday by a mob of their masked security enforcers was the "final showdown" in its efforts to oust the government.
Protesters wearing black and with their faces covered forced the National Broadcasting Services of Thailand (NBT) off the air.
The state broadcaster was off for several hours and the attack caused minor damage but no injuries, Thai media reported.
Police arrested about 80 people, some of whom were armed with guns, knives, sling shots and golf clubs, NBT reported.
The PAD initially denied any involvement in the raid but after the arrests PAD leaders at a semi-permanent protest venue in central Bangkok called on their followers to march on the state broadcaster.