Australian troops patrolled the waterfront in Dili on Tuesday, while convoys of armoured personnel carriers rattled through the streets.
Many East Timorese blame recent violence in the capital on Mari Alkatiri's decision to fire 600 soldiers in March, and some allege that he formed a hit squad to kill his political opponents, a charge he denies.
Protest organisers said they expected 1,000 people to turn out for Tuesday's rally with numbers anticipated to be even as high as 30,000 in coming days, as word of the anti-government demonstration spreads to the countryside.
Augusto Junior Tridade of the National Youth Forum speaking to hundreds of cheering young men who were among the first to turn out in front of the Government Palace, said: "We plan to stay here as long as it takes for Mari Alkatiri to go.
"He is the mastermind of all this violence," Tridade said.
"He ordered soldiers to kill civilian people and created a lot of problems in East Timor."
Alkatiri has come under criticism over his handling of the violence enveloping his nation since its break for independence from Indonesia in 1999, when revenge-seeking militias went on a deadly rampage.
His dismissal of more than 40% of the country's 1,400-strong armed force caused clashes with loyalist forces that gave way to gang warfare, with machete-wielding youths torching houses and looting government buildings.
Alkatiri's dismissal of 600
soldiers started clashes
At least 30 people have been killed since late April, and nearly 150,000 others have fled from their homes, though the violence has eased in recent weeks with the arrival of a 2,700-strong Australian-led multinational force.
Nino Pereira, a popular singer and lecturer at the National University, speaking to the Associated Press after addressing the crowd, said: "Mari Alkatiri came to power because of the people, so it is our right to take back that power.
"This is people power."
Tuesday's protest came amid claims by Vincente "Railos" da Concecao, an alleged hit-squad leader, that Alkatiri recruited and armed his men last month to eliminate his enemies.
The prime minister has denied the charge, which supporters fear could turn his ruling party against him and lead to his overthrow in a vote of no confidence in parliament.
Demonstrators erected banners in the seaside capital, most of them accusing Alkatiri - who spent most of the 24 years of Indonesian occupation of East Timor in exile in Marxist Mozambique - of being a communist.
They also handed out fliers linking Alkatiri, a Muslim in the mostly Roman Catholic country and a descendent of Yemeni immigrants, with al-Qaeda leader Osama bin Laden.
"We hate Mari Alkatiri. He is (a) descendent of bin Laden and he is (a) terrorist and communist," the flier said, also describing him as a "murderer" and "not pure Timorese."