Asked during a National Press Club appearance on Thursday about indications of a generally deteriorating relationship between Washington and parts of Latin America, Rumsfeld said he believes such a characterisation "misses the mark".
He said the rise of elected populist leaders in Latin America such as Chavez, Venezuela's president, was "worrisome".
"You've got Chavez in Venezuela with a lot of oil money," he said.
"He's a person who was elected legally just as Adolf Hitler was elected legally.
"And then (he) has consolidated power. And now, of course, is working closely with Fidel Castro and Mr Morales and others," Rumsfeld said, said referring to Evo Morales, Bolivia's new president, and Castro of Cuba.
Rumsfeld's comments were followed a day later by the US expulsion of Jeny Figueredo Frias, a chief of staff to the Venezuelan ambassador.
Frias was declared persona non grata and given 72 hours to leave the US.
Sean McCormack, the US State Department spokesman, said the US action was a direct response to Venezuela's expulsion of John Correa, the US navy attaché in Caracas.
"We don't like to get into tit-for-tat games with the Venezuelan government like this, but they initiated this and we were forced to respond"
US State Department spokesman
"We don't like to get into tit-for-tat games with the Venezuelan government like this, but they initiated this and we were forced to respond," he said.
On Thursday, Chavez accused Correa of spying.
Chavez said: "John Correa has to leave the country immediately.
"We have decided, in diplomatic terms, to declare him persona non grata, in plain Spanish that means to throw out of the country an officer at the US military mission for spying."
The socialist president, a critic of the United States, added that if other military attaches were caught spying, they could be detained.
Correa is alleged to have passed secret information from Venezuelan military officers to the Pentagon.
In Washington, Lieutenant-Colonel Mark Ballesteros, a Defence Department spokesman, said Correa "has been rotated back to the US mainland for further duties as assigned".
The US State Department denied that the attaché had been involved in spying.
Chavez accuses Washington of
trying to kill him
Janelle Hironimus, a State Department spokeswoman, said: "None of the attachés was or is engaged in inappropriate activities."
There have been increasing signs of hostility between Washington and Caracas.
Chavez refers to Bush as "Mr Danger," and is known to attack his "imperialist" policies. He also accuses Washington of trying to kill him.
US officials say Chavez has teamed up with Cuba to destabilise the region with his revolutionary message.