Gates said he intends to travel to Iraq soon to hear the "unvarnished" views of US commanders on how to improve the situation.
George Bush, the US president, said Gates will bring a "fresh perspective" on Iraq, acknowledging the current strategy was not working.
Gates succeeds Rumsfeld as the US military feels the strain of the deaths at least 3,000 troops over five years of war in Afghanistan and Iraq.
Rumsfeld was the chief architect of the war in Iraq and has repeatedly defended the decision to invade in March 2003.
Gates' appointment comes at a crucial juncture in the Iraq war - a conflict that cost Rumsfeld his job.
Prior to his nomination, Gates was a member of the Iraq Study Group, a a commission that recommended policy changes in Iraq to George Bush, the US president and the congress.
Beyond Iraq, Gates will advise the president on North Korea and Iran.
Bush nominated Gates, a CIA director during the administration of Bush's father, after voters swept Republicans out of power in an election driven in large part by frustration over the Iraq war.