This is according to an ABC News/Washington Post poll released on Monday.
Fifty-six per cent of those questioned - a new high - said the cost of the war outweighed the benefits and was not worth it. It marked a gain of seven percentage points from a poll conducted in July.
Fifty-seven per cent said they disapproved of the way Bush was handling the situation in Iraq, and 53% disapproved of the way Rumsfeld was handling his job, according to the survey.
However, 60% said the Iraqi elections scheduled for 30 January should go forward regardless of the security situation.
The poll also found that most Americans - 58% - still say US forces should remain in Iraq until order is restored.
Several Republican lawmakers have expressed doubts about Rumsfeld's performance, and some Democrats want him fired over his handling of the war.
More than half of the poll respondents - 52% - said Rumsfeld should be replaced.
Rumsfeld resignation call
Separately, a new USA Today/CNN/Gallup Poll also found growing pessimism over Iraq.
Bush said Rumsfeld was doing a
really fine job
Nearly 60% said they disapproved of the way the US was handling the situation in Iraq, and 52% said Rumsfeld should resign.
Bush defended Rumsfeld at a news conference on Monday, saying the Pentagon chief was "a good human being" and "doing a really fine job".
According to the ABC News/Washington Post poll, Bush's approval rating on Iraq dropped to 42%, down from 60% a year ago.
Fewer than half said they believed there had been significant progress towards restoring order in Iraq, down from 51% last summer.
The ABC News/Washington Post telephone poll of 1004 adults was conducted from 16-19 December and has a three-point margin of error.
The USA Today/CNN/Gallup poll of 1002 adults was conducted from 17-19 December and has a 4.5-point margin of error.