He will now oversee the US war in Iraq at a time when American troop levels are being reduced.
"He [Odierno] knows that we are at a pivotal moment, where progress remains fragile and caution should be the order of the day," Gates said at the ceremony, held in the halls - now a military base - of one of the palaces of Saddam Hussein, the former Iraqi president ousted in the US invasion of Iraq in 2003.
Odierno said that despite the fall in violence in Iraq - which Washington attributes to its so-called "surge" strategy - any gains could be reversed.
"Iraq is now a different country from the one I had seen first. However, we must realise that these gains are fragile and reversible," Odierno said.
But Odierno has been criticised for running roughshod over civilians during his first tour to Iraq in 2003-2004.
Some say Odierno's unit, the 4th Infantry division - which is in fact an artillery unit - used a heavy-handed strategy that has been accused of devastating large parts of the Iraqi population.
It was Odierno's soldiers who captured Saddam in December 2003.
But later, he appeared to have adopted Petraeus's approach.
From December 2006 to March 2008, Odierno implemented the US "surge" strategy, which Gates said made him the right person to replace Petraeus.
When he ended his second tour in March, Odierno said: "Just as important as the surge was the change in our tactics, techniques and procedures that got us back out in the neighbourhoods."
He has argued consistently against sharp cutbacks in troop levels in Iraq, which is negotiating a controversial security pact with Washington to determine troop levels that will remain in Iraq after a UN mandate expires at the end of the year.
Iraq currently handles security in 11 of its 18 provinces with plans to take over a couple more by the end of the year.
Petraeus will become the new chief of US central command with responsibility for US troops from the Horn of Africa to Central Asia, including overall charge of the conflicts in both Afghanistan and Iraq.
Iraq was spiralling into all-out civil war when Petraeus took over as commander in February 2007, almost four years after Saddam was toppled from power.
Violence in Iraq lessened last year and has now fallen to a four year low, something many analysts have attributed to the counter-insurgency strategies of Petraeus.