"He's been saying that forever," he told Al Jazeera.
"Mr Rajapaksa has proven to be a very opressive and discriminating president. The Tamils are not celebrating today as an independece day. Rajapkasa was not voted in in the north and east, where the Tamils are - it's their land."
Rajapaksa's speech in Kandy was his first public address to the nation since his re-election as president in January, but comes amid continued wrangling by General Sarath Fonseka, his former army chief and election rival.
Ahead of the independence day celebrations, about 5,000 of Fonseka's supporters took to the streets of Colombo, the capital, to protest against the election results.
Fonseka told the crowd that "Rajapaksa has grabbed and stolen your right to elect a president".
During his speech he held up what he said was a burnt ballot paper, part of several bundles of ballots the opposition said were marked in his favour but were discovered discarded.
"We will get your rights at any cost. We will not allow them to celebrate the rigged victory," he said.
Dayananda Dissanayake, the country's elections commissioner, rejected the fraud allegations, though he said that state media violated voting guidelines when he announced the results on election day.
"I am not satisfied with what has happened in the campaign period," Dissanayake said.
"But I stand by the voting process and the results."
Local and international election observers generally praised the election, noting the campaign period was violent - five people were killed - but said polling day went smoothly aside from a few minor incidents of malpractice.
Meanwhile Rajapaksa appears to be consolidating power ahead of parliamentary elections in April.
The Sri Lankan supreme court ruled on Wednesday that his new six-year term will begin in November, in effect allowing Rajapaksa, who called elections two years early, to rule for the next seven years.
Rajapaksa's government has been accused of orchestrating a crackdown on the media after a series of websites were blocked and at least one reporter detained. One journalist is said to be missing, one has been assaulted and others have received death threats.
|Thousands of opposition supporters took to the streets of Colombo on Wednesday [Reuters]
"Now that the president has been re-elected, there appears to be a settling of scores with critics of the government," said Brad Adams, Asia director of the New York-based Human Rights Watch (HRW) group.
"Just days after the election, some officials seem to be on a campaign to abuse their power."
Rajapaksa has also dismissed at least 14 senior military officers – believed to be Fonseka loyalists - who the defence ministry said were a "direct threat to national security" after the presidential elections.
The state-run Daily News on Wednesday reported that a total of 37 people have been arrested so far, among them 15 former military members of Fonseka's staff who were arrested when police commandos raided his office on Saturday.
Rajapaksa also transferred dozens of officers and promoted several considered loyal to his administration.
Since the day after the election, the government has made increasing allegations that Fonseka and his supporters may be plotting a coup and assassination attempt on the president and his relatives.
Fonseka has vowed to challenge the election results in court, but has not yet done so.
"If [Rajapaksa] won genuinely, why is he so scared of me? Why are they not allowing us to move freely. Why is he arresting my supporters?" the former army chief said at Wednesday's rally.