The Israelite Federation in Rio de Janeiro state and the Simon Wiesenthal Center, a prominent international Jewish rights group, had objected to the float.
"A guy dressed as Hitler atop a pile of dead Jews, that's too much. It's beyond the limits of common sense," said Sergio Niskier, president of the Israelite Federation.
"There is no way we could accept that."
On Wednesday, another samba group agreed to remove the swastika symbol from its dancers' uniforms after complaints from the federation.
The Rio Carnival, which takes place over the weekend, is famous for parades by samba schools with colourful floats and costumes and street parties where costumed revellers drink and dance all night.
The elaborately decorated floats are a key part of each samba school's presentation, along with thousands of dancers and drummers led by near naked carnival queens.
Viradouro's parade theme this year is "Shockers" and it also reportedly includes floats depicting the shock of birth, the shock of horror and the shock of cold.
In previous carnivals samba schools have had problems with the Catholic church over floats displaying religious references.
An estimated six million Jews perished in the Holocaust, instigated by Germany's Nazi government under Adolf Hitler, in World War II, along with hundreds of thousands of gypsies, Catholics and other groups.