Rumsfeld also told a Pentagon news conference on Tuesday that prisoners at the US naval base at Guantanamo Bay, Cuba, were staging a hunger strike that began in early August as a ploy to attract media attention.
The three UN investigators, including one who focuses on torture, said on Monday that they would turn down an invitation extended by the Pentagon on Friday to visit Guantanamo unless they were permitted to interview the detainees. The invitation came nearly four years after the visits were first requested.
No policy change
Rumsfeld said the US government will not change its policy of giving such access to detainees only to the International Committee of the Red Cross, a neutral body that keeps its findings confidential.
"There's got to be a limit to how one does that," Rumsfeld said of providing access to detainees.
"And the ICRC has been doing it for a great many years and has had complete and total access ever since Guantanamo was opened. And so we're not inclined to add (to) the number of people that would be given that extensive access."
The invitations went to Austria's Manfred Nowak, special investigator on torture; Pakistan's Asma Jahangir, who focuses on religious freedom; and Algeria's Leila Zerrougui, who looks into arbitrary detention.
Human rights activists have criticised the United States for the indefinite detention of about 500 prisoners at Guantanamo.
The Guantanamo detentions
have sparked worldwide protests
Former prisoners have stated they were tortured there, and the ICRC last year accused the US military of using tactics "tantamount to torture" on Guantanamo prisoners. The military has denied torture has occurred.
The UN investigators said they proposed a 6 December visit but would go only if permitted to talk to the prisoners.
Zerrougui said on Monday that the UN investigators had never agreed to visit a place where they would not have full access to all detainees and asked the United States to provide such access "in the spirit of compromise".
The US military said on Tuesday that 27 detainees were engaging in the hunger strike, including 24 receiving forced-feedings. Detainees' lawyers estimated that about 200 are taking part.
Asked about the motivation of the hunger strikers, Rumsfeld said: "Well, I suppose that what they're trying to do is to capture press attention, obviously, and they've succeeded."
US District Court Judge Gladys Kessler last week ordered the government to provide medical records on Guantanamo prisoners who are being force-fed and to notify their lawyers about forced feedings.
The judge said detainees' lawyers had presented "deeply troubling" allegations of US personnel violently shoving feeding tubes as thick as a finger through the men's noses and into their stomachs without anesthesia or sedatives, with detainees vomiting blood as US personnel mocked them.
Rumsfeld appeared to distance himself from the decision to force-feed detainees.
"I'm not a doctor and I'm not the kind of a person who would be in a position to approve or disapprove. It seems to me, looking at it from this distance, is that the responsible people are the combatant commanders. And the Army is the executive agent for detainees," Rumsfeld said.