They said going on the planned 6 December trip without being able to speak privately to inmates "would have created a disastrous precedent".
"Since the Americans have not accepted the minimum requirements for such a visit, we must cancel," Manfred Nowak, the UN envoy in charge of investigating torture allegations around the world, said on Friday.
"It means there will be no visit this year for the report we are preparing, but naturally we are totally prepared to go to Guantanamo in the future if the Americans assure us of their full cooperation."
The 6 December date had been set after more than three years of discussions between US and UN officials amid claims of human rights abuses at the camp.
"Under the circumstances, we will not be travelling to Guantanamo," the five UN experts, including Nowak, said in a statement earlier in Geneva.
Protests against the Guantanamo
detentions have been growing
"Doing so would undermine the principles of UN human rights fact-finding missions," the statement added.
"It is particularly disappointing that the United States government, which has consistently declared its commitment to the principles of independence and objectivity of the fact-finding mechanisms, was not in a position to accept these terms."
The United States on Tuesday refused the UN human rights monitors' demand for an unconditional inspection of the Guantanamo camp which they had said was standard practice for a "credible, objective and fair assessment".
The US State Department said Washington was open to inspectors but slammed the way the UN experts had piled up public pressure for access.
The United States provided regular access to the International Committee of the Red Cross (ICRC) and had consulted with governments on the cases of their nationals detained there, the US State Department said.
The UN experts maintain that the ICRC's monitoring is very different.
The ICRC as a matter of policy does not make its findings on humanitarian conditions public in order to preserve access to prisons that may otherwise be closed to them.
"Since the Americans have not accepted the minimum requirements for such a visit, we must cancel"
The UN experts, in contrast, are mandated to investigate allegations of human rights breaches and report publicly to the UN General Assembly and the world body's top watchdog, the Human Rights Commission.
The US government has been sharply criticised for conditions at Guantanamo, where 520 detainees are held without trial and where some have gone on hunger strikes.
Most of those held there were captured after a US-led offensive toppled the Taliban government in Afghanistan in late 2001.
The United States has declared the detainees to be illegal enemy combatants not protected by the Geneva Conventions.
The UN monitors had already reluctantly agreed to limit the inspection to a single day instead of three and to send three investigators instead of five.
Even without a visit, they still intend to write a report on conditions at the prison based on eyewitness accounts from released detainees, meetings with lawyers and information from human rights groups.