Around 200 Bakiyev supporters, some with automatic rifles, were holed up in a government building they had seized the previous evening.
About 4,000 backers of the Ata-Meken party that supports the interim government arrived early on Friday to try to evict the occupiers, but many dispersed when gunfire broke out, leaving a crowd of several hundred.
Some men in the approaching mob returned fire, while others fought with sticks.
In the capital Bishkek, Roza Otunbayeva, the head of the interim government, said masked "mercenaries" were holed up inside the Jalal'abad administration building.
"Our tactics are to take them alive," she said.
"Our forces are approaching. Some are there," Otunbayeva said.
In the city of Osh and the the town of Batken, supporters of the interim government had managed to retake regional administration buildings.
Al Jazeera's Robin Forestier-Walker, reporting from Bishkek, said: "Teyit [Bakiyev's home village] was attacked by an unknown gang and five houses [reportedly Bakiyev owned] set alight.
"The provincial mayor in Jalal'abad has called for citizen vigilante groups to form to protect citizens and businesses".
Azimbek Beknazarov, a senior interim government official, said there was information that Bakiyev supporters were planning mass protests against the authorities on May 17.
"Popular militia groups are forming throughout the country with the aim of thwarting a destabilisation of the country," he said.
He also said "south of the country is completely under the control of the authorities" and that the government had ordered the arrest of the organisers of the recent unrest.
Forestier-Walker said that when he visited the south after Bakiyev's overthrow, there was a very limited presence of security forces.
'You don't need very many people to destabilise the situation," he said.
"You just need to be armed and determined. That seems to be the case in Jalal'abad for the moment, it's a very tense situation..
"The question is whether yesterday's and today's disturbances are a flash in the pan or a sign of things to come."
Interim authorities said Bakiyev, who fled the country after an uprising last month, was behind the unrest and said its organisers "wanted to ignite an inter-ethnic conflict in the country".
"Bakiyev is behind all this," Omurbek Tekebayev, the deputy chairman of the interim government, said on state television.
The south of Kyrgyzstan was the power base of Bakiyev, who has now taken refuge in Belarus.
On Thursday, his supporters peacefully seized regional administration buildings in Osh, Jalal'abad and Batken.
They also occupied the airports in Osh and Jalal'abad.
Otunbayeva, took power in April after anti-government protests sparked clashes with security forces leaving at least 86 people dead.
Bakiyev, who fled the capital, himself came to power in a popular uprising, the so-called Tulip Revolution of 2005, but became increasingly unpopular amid allegations of corruption and mismanagement.