They wanted Ban to pursue the UN Security Council's call in October 2007 for Myanmar to release the prisoners, including Aung San Suu Kyi, the opposition leader and Nobel Peace laureate.
Michele Montas, a spokeswoman for Ban, said the UN chief "said he would like to visit Myanmar again to discuss a broad range of issues but that he will not be able to do so without reasonable expectations of a meaningful outcome".
"[Ban] said he would like to visit Myanmar again to discuss a broad range of issues but that he will not be able to do so without reasonable expectations of a meaningful outcome"
Michele Montas, spokeswoman for Ban
She added that Ibrahim Gambari, Ban's special envoy to Myanmar, would also not return to the country formerly known as Burma, until he had a "real possibility of moving forward there".
Ban travelled to Myanmar in May in the wake of Cyclone Nargis, persuading Senior General Than Shwe, the country's leader, to ease access for foreign aid workers and relief supplies.
On Wednesday Ban received the letter and spoke with Kjell Magne Bondevik, the former Norwegian prime minister, who led the effort.
The letter was released by Bondevik's Oslo Centre for Peace and Human Rights and by Freedom Now, a Washington-based advocacy group.
"We urge you to make it clear that all political prisoners in Burma must be released by the end of this year, regardless of whether you travel to Burma," the letter said.
Myanmar's military, which has held power since 1962, has ramped up its crackdown on government critics since Buddhist monks led anti-government protests in September last year.
International human rights groups say the ruling generals are holding more than 2,100 political prisoners, a sharp increase from pre-demonstration figures of nearly 1,200 in June 2007.