Nearly four years after the visits were first requested, Washington said on Friday the three envoys, including the UN rapporteur on torture, could visit foreign terrorism suspects because it had "nothing to hide".
But although they could question US military officials, the envoys would not be allowed to speak to any of the some 505 detainees, the Pentagon said, adding that was the role of the International Committee of the Red Cross.
In a statement, the three envoys said on Monday: "We cannot accept the exclusion of private interviews with detainees...this would not only contravene the terms of reference for fact-finding missions ... but also undermine the purpose of an objective and fair assessment."
Human rights activists have criticised the United States for the indefinite detention without trial of the detainees at the US naval base at Guantanamo Bay, Cuba.
The invitations were extended to Austria's Manfred Nowak, special investigator for the United Nations on torture, Pakistan's Asma Jahangir, who focuses on religious freedom, and Algeria's Leila Zerrougui, who looks into arbitrary detention.
The envoys also expressed disappointment that two others - Leandro Despouy, special investigator on the independence of judges and lawyers and Paul Hunt, special rapporteur on mental and physical health - had not been included.
They also felt that the one day granted them by the Pentagon for the visit was insufficient.
Nevertheless, the envoys said that they had "carefully considered the invitation and decided to accept it" conditionally on the interviews taking place.
They added that they were confident that the US government, "which attaches great importance to the principles of independent and objective fact-finding", would understand their position and accept their demand.
"We cannot accept the exclusion of private interviews with detainees...this would not only contravene the terms of reference for fact-finding missions ... but also undermine the purpose of an objective and fair assessment"
Statement issued by the three UN envoys invited to visit Guantanamo
Criticism by human rights groups has escalated in recent weeks with the US military's disclosure that it was force-feeding Guantanamo detainees staging a hunger strike over their conditions and lack of legal rights.
The Pentagon has defended its treatment of prisoners and denied that torture has occurred at the Guantanamo facility, which opened in January 2002, just months after the September 11, 2001, attacks on the United States. Most of the detainees were seized in Afghanistan.