The former members of the Liberation Tigers of Tamil Eelam (LTTE) were expected to win the elections which are being seen as a test of a plan for greater devolution in minority Tamil areas.
Batticaloa is a Tamil majority area.
One municipal council and eight local government bodies will be elected.
6,500 security personnel deployed during polling.
UNP, the main opposition, and Tiger-backed TNA have boycotted the election.
The district was under Tamil Tiger control for 13 years.
Government troops recaptured the area last July.
However, human rights groups believe the elections are unlikely to be free and fair because of violence and intimidation.
They have also questioned the government's decision to endorse the TMVP, even though it is now working with the government in Batticaloa.
They have been accused of widespread abuses, including recruiting child soldiers, abductions and killings.
Karuna himself left Sri Lanka in 2007 and is now in the UK.
Edwin Krishnanandaraja, also known as Pradeep Master, a political leader of the newly registered party in Batticaloa, is a former Tamil Tiger who joined the rebels as a child soldier.
He is tipped to be Batticaloa's next mayor, and rejects the accusations of abuses by his group.
"They are baseless," Krishnanandaraja said. "Our members were being killed. Because of that, we used weapons. When our political right is confirmed, we will hand them over."
Rajapaksa has refused to disarm the TMVP, arguing that it could not find weapons on its members.
Rights groups have also accused the state of complicity in the abuses and also believe polls have come too soon.
"In Batticaloa, not only TMVP, many other armed groups are also there," Kingsley Rodrigo, chairman of the People's Alliance for Free and Fair Elections, the island's main election monitoring body, said.
|Monday's blast occurred in a mostly Tamil |
neighbourhood in Colombo [Reuters]
"They are not using arms to campaign, but they have been keeping the arms with them. So I am not going to say this election is a free and fair one."
However, Palitha Kohona, the Sri Lankan foreign affairs secretary, told Al Jazeera that the local elections are a step forward for the area.
"The government has made its promise, it promised to have elections so that the people could elect their own representatives who will look after their needs in the circles of power," he said.
"It's never too soon to introduce democracy. It was only after the LTTE took control of certain parts of the east and of the north that democracy became a distant dream."
A host of other former armed groups who joined the democratic mainstream in the 1980s are taking part in the poll, as well as the island's main Muslim party.
But with the main opposition UNP and Tiger-backed TNA boycotting the poll, some say that the election is a farce.
Sinnakutty Nagalingam, a father of three who works as a mason, said: "We don't expect the leaders to give us anything other than a chance to live in peace. If there is peace, we can work and take care of ourselves.
"With this situation, if we go out to work, there is no certainty that we will return home."
Hospital authorities say that one person was killed and several wounded, including four school children, in the bomb blast in Colombo's Wellawatta neighbourhood.
The bomb was hidden in a flowerbed on the main Galle highway, Udaya Nanayakkara, a military spokesman, said.
Officials say the attack bears all the hallmarks of the LTTE, but there was no immediate claim of responsibility from the group.
The government has become increasingly isolated in the international community after it formally scrapped a six year truce.
Since then it has launched an offensive against the Tamil Tiger group, which is seeking an autonomous Tamil nation, and has said it plans to defeat the group by the end of the year.