Anaya's comments came as Australia launched its latest bid to address inequality, ill-health and poverty among its 500,000 indigenous people, and announced it will form a new national representative body this year to advise it on related policies.
But on Friday the Australian government rejected Anaya's harsh criticisms, who said the aboriginal minority still suffers from "entrenched racism".
Jenny Macklin, the Australian indigenous affairs minister, said she knows the programme was controversial and even acknowledged that some parts of the crackdown two years ago were contentious.
"We as a government and as a country have to confront the realities of indigenous people, particularly in remote parts of Australia," she said.
Macklin said the government's priority was to protect the rights of indigenous children who are subject to high levels of abuse, and to do something to fix the Third World living conditions faced by many of the Aborigines.
"Stigmatising aboriginal people further will not help aboriginal children's health and education outcomes or improve safety in communities"
Rodney Dillon, Amnesty International
"Many of these families tell me it's putting food on the table for their kids that used to be spent on grog [alcohol]," said Macklin. "I don't think anyone could disagree that that isn't a good thing."
Aborigines make up 2.5 per cent of Australia's 22 million population and suffer disproportionate rates of infant mortality, health problems and suicide.
Despite pumping billions of dollars into community programmes, housing and education in recent decades, the Aborigines remain the poorest, unhealthiest and most disadvantaged group with an average life span 17 years shorter than other Australians.
In 2007 John Howard, the former Australian prime minister, suspended the Racial Discrimination Act in the Northern Territory as he sent troops and police to help curb alcohol-fuelled sex abuse and domestic violence in isolated desert communities.
His successor Kevin Rudd has refused to scrap the policy since taking office, despite issuing a historic apology for the wrongs suffered since white settlement in 1788.
In backing Anaya's report on intervention programmes, Amnesty International, the London-based human rights watchdog, said Australia was stigmatising the aboriginal communities and failing to fulfil its human rights obligations.
Rodney Dillon, the group's indigenous-rights advocate, echoed the UN's concerns and said the implementation of the policy in the Northern Territory "does not fulfil Australia's international human rights obligations".
"It's the government's responsibility to provide health care and education to all its citizens," he said.
"Stigmatising aboriginal people further will not help aboriginal children's health and education outcomes or improve safety in communities."