The government gave municipal authorities permission on Wednesday to dig up 1,400 graves at a Muslim cemetery in the Ilala area.
Work started straight away and the whole process is expected to take at least two weeks.
The bodies will be reburied in Segera town 18 km west of the Indian Ocean city.
Residents who were made to move the bodies of family and friends accused the government of targeting the Muslim
"This is disrespectful to the dead"
Sheikh Ali Kileo
"It seems like another government move to frustrate us," said one resident as he carried the bones of his relative away from the site, wrapped in a white sheet.
And Tanzanian imam Sheikh Ali Kileo said proper religious rites had not been carried out when exhuming the remains.
"This is disrespectful to the dead," he said.
He added that he would ask Muslims to hold a demonstration after prayers on Friday.
Officials said the government would spend about 142 million shillings ($136,000) on the project, with each family receiving 60,000 shillings ($57) per grave.
Muslims make up about a third of the population in Tanzania.