Rights advocates say migrants who were forced to leave the mainland lack healthcare, education and food.
At least 27 Rohingya refugees have been missing after their boat sank during an attempt to escape a Bangladeshi island camp criticised by human rights groups, officials said.
After the incident on Saturday, the United Nations refugee agency said it “remains concerned about reports of refugees being arrested and detained for attempting to leave Bhashan Char”.
Nearly 20,000 Rohingya have been relocated to Bhasan Char island, which takes the full force of cyclones that roar across the Bay of Bengal each year.
Bangladesh eventually wants to rehouse 100,000 of its approximately one million Rohingya refugees to the island, moving them from cramped settlements on the mainland.
But some Rohingya say they were coerced into relocating, while thousands protested against living conditions on the island in June.
Police said the small fishing boat sank with 41 Rohingya on board after encountering rough weather in the Bay of Bengal near the island.
“Refugees were on board the boat when it capsized late last night. Some 14 of them have returned unharmed,” Mahe Alam, the police chief on the Bhasan Char island, told Anadolu Agency over the telephone.
He added a joint rescue operation conducted by police, coastguard, navy and air force is still ongoing to reach out to the missing ones.
Sujit Kumar Chanda, the government administrator for Bhasan Char, told the AFP news agency that the missing included women and children.
Chanda said a two-year-old was dramatically saved by his father who held the child above water while swimming for safety.
Bangladesh started relocating refugees to the controversial island camp in November, saying the refugee camps in the country’s southeast were overcrowded.
Police said many Rohingya have fled the island in recent months and been arrested in coastal towns in Bangladesh’s Chittagong region.
Abdur Rahim, 27, who fled Bhashan Char months ago, said a lack of work on the island has pushed people to flee.
“Living there is horrible. There is no opportunity to meet relatives. It’s like they are keeping us in jail without giving us any opportunities,” he told AFP.
Amnesty International’s South Asia campaigner Saad Hammadi said Bangladesh must “ensure that any relocation to the island is voluntary” and refugees have “the right to freedom of movement between the island and mainland, where many of them have their families and relatives”.