The opposition Australian Labor Party said on Thursday Robert Hill should go because he had misled the public about Australia's knowledge of the scandal.
A Labor defence spokesman, Chris Evans, said an explanatory statement by the minister to the Senate was inadequate.
Until Wednesday, Hill had mistakenly maintained that Australia knew nothing about the abuse until it was publicly revealed in April.
However, department staffers had reports raising concerns about Abu Ghraib dating back to June 2003.
"No one's been sacked, no one's resigned, no one's admitted any blame," Evans said. "They've sought to blame everybody else but themselves and that's not good enough.
"The buck stops with senator Hill and he ought to go."
Labor senator John Faulkner said Hill's explanation merely restated what was already on the public record.
"It's a damp squib, it's not an explanation, it's a complete whitewash," he said.
The measured Australian Financial Review newspaper also lashed out at Hill, criticising his "breathtaking contempt for any obligation to be accountable to voters".
The Sydney Morning Herald said the government's argument that no Australians were involved in the abuse by US forces at Abu Ghraib was a "smoke and mirrors" diversion that deliberately missed the point.
Robert Hill has dismissed calls for
him to quit as a 'political game'
"We cannot accept an ally's rationale for prisoner abuse, then condemn foes for similar actions," it said.
"The issue has never been about what individual Australians did or didn't do to prisoners in Iraq - it's been about what Australia, an occupying power, did and didn't do."
Hill dismissed calls for his resignation as a "political game" and said he would not order a review into why his department failed to pass on its concerns.
"I forgive a mistake, in the background where the department as a whole has worked so well over the past few years," he said.