"I've seen four people who were injured - some of them may be dead. But the crowd has been largely chased away.
"A lot of them were marched out of the tombs compound with their hands up and sent down the road, chased off with tear gas."
'Attack on Buganda'
Buganda is one of Uganda's four historic kingdoms and the Baganda are the biggest ethnic group in the country.
Kasubi, designated as a world heritage site by Unesco, the UN cultural agency, is an important tourist site housing the burial grounds of four former kings of Buganda. The last king was buried there in 1971.
Baganda have previously complained that the government has tried to expropriate the kingdom's land and have been involved in disputes over the influence of Buganda's traditional leaders.
A spokesman for the Buganda kingdom on Wednesday described the fire as "an attack on Buganda", whose people are concentrated in the south of Uganda and Kampala.
"This fire is very strange given what we [the Baganda] have been going through," Peter Mayiga said without giving further details.
There was no immediate confirmation that the blaze was started deliberately and Kale Kayihura, the chief of the Uganda police, said his officers were still investigating the cause of the fire.
He also said that firefighters had been prevented from reaching the site in time to put out the blaze before it gutted the burial site late on Tuesday.
"When the fire broke out, police were called in and got there in time but the fire brigade was obstructed by a hostile crowd, three trucks were damaged and a fireman injured," Kayihura told the AFP news agency.
"Faced with this hostility and in an effort to stop the fire from destroying the tombs, the officer fired some shots in the air to disperse the crowd but no one was hurt," he said.
The violence triggered fears of renewed tension between the government and ethnic Baganda, who accuse the government of undermining their kingdom.
Last September, riots in Kampala left at least 17 people dead after the government prevented Ronald Muwenda Mutebi, the current Buganda king, from visiting a district near the capital.
The government said the district had its own traditional ruler and it did not owe allegiance to the king.
The king holds a largely ceremonial position in Uganda, but wields considerable influence among his people.
Monarchies were restored in Uganda in 1993 after they were banned in the 1960s, but they are not allowed to participate in elective politics by campaigning or fielding candidates.
The kingdom's radio station - the Central Broadcasting Service (CBS) - remains banned after it was taken off air in the aftermath of the riots, its staff accused of inciting violence.