Howard, who held a landmark meeting with Muslim leaders last week to secure their support for joint action against Islamic extremism, said on Monday he did not support such a ban.
"If you ban a headscarf you might, for consistency's sake, have to ban a yarmulke or a turban," he told reporters.
"It does become rather difficult and rather impractical."
Two backbenchers in Howard's Liberal Party had called for the ban with one, Bronwyn Bishop, describing headscarves in public schools as "an iconic item of defiance".
"It's not just a headscarf, it is a challenge to our freedoms and way of life," she said.
Last week, Education Minister Brendan Nelson told reporters that Muslims who did not want to accept "Australian values" should "clear off".
"If you ban a headscarf you might, for consistency's sake, have to ban a yarmulke or a turban"
Australian prime minister
But he has refused to support the call for a ban on headscarves.
The debate echoes that in several European countries.
France last year prohibited the wearing of conspicuous religious insignia in state schools, including Muslim headscarves, Jewish yarmulkes, Sikh turbans and large Christian crosses.
Australia's small Islamic community has been in the spotlight since British-born Muslims killed themselves and 52 other people in attacks in London in July, with Howard suggesting that similar attacks could take place in his country.