Speaking at a White House press conference on Wednesday, George Bush, the president, said Rumsfeld was a patriot who had "served his country with honour and distinction" but recognised that a change was needed at the Pentagon.
"After a series of thoughtful conversations, secretary Rumsfeld and I agreed that the timing is right for new leadership," Bush said.
He conceded that his Iraq policy was "not working well enough, fast enough" and said he took responsibility for a "thumpin' " by Democrats in the congressional elections.
He said: "I thought we were going to do fine yesterday ... it shows what I know."
The Democrats wrested control of the US House of Representatives from the Republicans and stand on the brink of also winning the Senate, pending close results from the state of Virginia, which would put the party in control of both houses for the first time in 12 years.
"I recognise that many Americans voted last night to register their displeasure with the lack of progress being made there [in Iraq]," Bush said.
Bush said he took responsibility for the Democratic election victory, acknowledging that there was widespread anger over Iraq among American voters.
"I'm obviously disappointed with the outcome of the election," Bush said.
"As the head of the Republican party, I share a large part of the responsibility."
Democrats wrested control of
the House of Representatives
Bush also made clear that Dick Cheney, his vice-president and another advocate of the Iraq war, would stay on until the end of his term in January 2009.
Bush said his party would now "work together" with the Democrats and independent parties to work on "the great issues facing this country".
Robert Gates, former director of the CIA, will take Rumsfeld's place, Bush said.
Nancy Pelosi, the Democrat representative of California and probable next leader of the House, said she welcomed the change in Pentagon leadership, saying: "I think it will give us a fresh start."
Rumsfeld, 73, steps down after almost six years as defence secretary.
A blunt and divisive figure who was one of Bush's closest advisers, he had been Pentagon chief since 2001, directing the invasions of Iraq and Afghanistan and undertaking the modernisation of the US military.
He also created the Guantanamo Bay jail for foreign terror suspects and presided over the defence department during the Abu Ghraib prisoner abuse scandal.
Gates (C) speaks after Bush
announced his appointment
While some have praised him for invigorating the US military, detractors have condemned him as a warmonger who mismanaged the war in Iraq and failed to foresee its current bloody outcome.
His resignation follows several calls for him to step down, the most recent only a few days ago when editorials published in four US military journals urged him to quit for "losing control" over the situation in Iraq.
In April 2006 he was also criticised by several retired army generals who said he had made strategic blunders in Iraq and disregarded military advice.