Protesters, many of whom have since withdrawn from the venue, said Abhisit's government had no right to host the summit, which brings together leaders from the 10-member Association of Southeast Asian Nations (Asean) plus China, Japan, South Korea, India, Australia and New Zealand.
"We will peacefully block the area and the summit site," Arisman Pongreungrong, a protest leader, told the Associated Press.
"We do not want any violence."
The so-called red-shirt protesters have already brought large parts of Bangkok to a standstill with three days of protests calling for Abhisit to stand down.
The red-shirts are supporters of Thaksin Shinawatra, the exiled former prime minister ousted in a bloodless 2006 coup.
They say the current prime minister came to power illegally and blame his associates for orchestrating Thaksin's removal.
They are demanding Abhisit stand down immediately and hold fresh elections.
Abhisit has rejected the protesters demands and has ordered increased security in Pattaya for the summit.
About 8,000 police have been deployed around the summit venue.
"Nothing untoward will happen to the leaders. We will make sure of that," Suthep Thaugsuban, the deputy prime minister, said seeking to assure visiting heads of state that they will be safe.
The summit is expected to focus on measures to combat the global economic crisis, stressing the importance of preserving free trade and fighting protectionism.
China and the Asean states are also due to sign a long-planned free-trade agreement, a key in the regional body's ambition to create the world's largest free-trade area.
|The red-shirts say they will not stop protests until Abhist quits [Reuters]
But the meeting has been criticised by human rights groups who say rights issues will again take a back seat, with Asean leaders sticking to their principle of non-interference in each another's internal affairs.
Al Jazeera's Tony Cheng, reporting form Pattaya, said: "The crowds are slowly dispersing, but most people anticipate that this will just be a break [in demonstrations] and that this is a very major problem for Thailand
"It is very damaging for the economy. Last year when another group of protesters, the yellow-shirts, closed down the international airport it was estimated to have cost close to a $1bn.
"Tourism and trade have been deeply affected. All the countries in this region that are meeting in Pattaya ... are all in deep [economic] trouble, and Thailand in particular with this kind of disruption is going to face more trouble than most."
In an apparent effort to ease tensions and keep the peace ahead of the summit, Abhisit declared Friday a public holiday.
But red-shirt protest leaders have said they see this as a sign the prime minister has been rattled by the demonstrations and have vowed to press ahead with demonstrations.
The protests are the biggest challenge yet to Abhisit's four-month-old government - the fourth administration since the coup in a long-running political crisis that revolves around Thaksin.
|Abhisit declared Friday a public holiday in an apparent effort to ease tensions [EPA]
The protesters say Thaksin was wrongfully ousted and Abhisit, who was appointed by parliament in December, took power illegitimately.
The protesters are angry about the way Abhisit took power after a court ruling that removed a government made up of Thaksin allies from power.
That ruling came after months of protests by rival, "yellow-shirt" protesters claiming allegiance to the monarchy.
Leaders of the yellow-shirt protests have said they are considering returning to the streets to challenge the current demonstrations – raising fears that there could be violent clashes.