They said Abhisit's government had no right to host the summit, which brings together leaders from the 10-member Association of Southeast Asian Nations (Asean) plus China, Japan, South Korea, India, Australia and New Zealand.
The summit is expected to focus on measures to combat the global economic crisis, stressing the importance of preserving free trade and fighting protectionism.
China and the Asean states are also due to sign a long-planned free-trade agreement, a key in the regional body's ambition to create the world's largest free-trade area.
The so-called red-shirt protesters have already brought large parts of Bangkok to a standstill with three days of demonstrations calling for Abhisit to stand down.
They are supporters of Thaksin Shinawatra, ousted as prime minister in a bloodless 2006 coup and currently in exile.
"We will peacefully block the area and the summit site," Arisman Pongreungrong, a protest leader, told the Associated Press.
"We do not want any violence."
They say Abhisit came to power illegally and blame his associates for orchestrating Thaksin's removal.
They are demanding Abhisit stand down immediately and hold fresh elections.
Abhisit, who has rejected those calls, said on Friday that the summit would continue as planned.
"We will ensure that the meeting will proceed smoothly. I would like to reiterate that we can provide security," he said.
About 8,000 police have been deployed around the summit venue.
"Nothing untoward will happen to the leaders. We will make sure of that," Suthep Thaugsuban, the deputy prime minister, said seeking to assure visiting heads of state that they will be safe.
But the meeting has been criticised by human-rights groups who say rights issues will again take a back seat, with Asean leaders sticking to their principle of non-interference in each another's internal affairs.
Damaging to economy
Al Jazeera's Tony Cheng, reporting form Pattaya, said the protests are very damaging for the economy.
"Last year when another group of protesters, the yellow-shirts, closed down the international airport it was estimated to have cost close to a $1bn," he said.
"Tourism and trade have been deeply affected. All the countries in this region that are meeting in Pattaya ... are all in deep [economic] trouble, and Thailand in particular with this kind of disruption is going to face more trouble than most."
|Abhisit declared Friday a public holiday in an apparent effort to ease tensions [EPA]
In an apparent effort to ease tensions and keep the peace, Abhisit declared Friday a public holiday.
But red-shirt protest leaders have said they see this as a sign Abhisit has been rattled by the demonstrations and have vowed to press ahead with demonstrations.
The protests are the biggest challenge yet to Abhisit's four-month-old government - the fourth administration since the coup in a long-running political crisis that revolves around Thaksin.
The protesters say Thaksin was wrongfully ousted and Abhisit, who was appointed by parliament in December, took power illegitimately.
The protesters are angry about the way Abhisit took power after a court ruling that removed a government made up of Thaksin allies from power.
That ruling came after months of protests by rival "yellow-shirt" protesters claiming allegiance to the monarchy.