A representative for the UN High Commissioner for Refugees said on Sunday she was "very worried" about the plight of the refugees who have been living in eastern Chad for three years.
There are apprehensions that Chad could banish the refugees from Darfur in Sudan in view of the escalating tensions between the two countries.
Ana Liria-Franch said: "The refugees have already suffered enough in Darfur. We do not want them to become victims of the tension between the two countries".
She also said the rebels were coming into the camps to recruit refugees as fighters, often under duress.
In February, 24 young men were recruited from the camp at Kounoungo, and in March several hundred, including boys as young as 14, joined the rebellion from the camps in Breidjing and Treguine , she said.
Idriss Deby, the president of Chad, threatened to close camps housing refugees from the Darfur region after rebels, he claims are backed by the neighbouring regime, attacked the Chadian capital N'Djamena on Thursday.
The non-governmental organisation Care, which is active in four of the 12 camps in eastern Chad, said it believed Deby was playing a political game and it was unlikely he would carry out his threat.
"The refugees have already suffered enough in Darfur. We do not want them to become victims of the tension between the two countries"
Representative for the UN High Commissioner for Refugees
Nicolas Palanque from Care said: "It does not seem realistic to relocate 200,000 refugees. Where would they go?"
He pointed out that Chad has for months been calling for a UN force to be deployed along its troubled border with Sudan and said Deby was now trying to force the hand of the international community.
"But it is not impossible that the government could decide to close the camps from one day to the next and the humanitarian fallout will be very serious," he warned.
Aid workers said the situation on the Chad-Sudan border has become increasingly dangerous since December as rebels infiltrate the camps which Chadian soldiers are supposed to guard.
"We started having serious problems in December," a representative from Care said, adding that rebels have stolen about 20 vehicles from humanitarian organisations.
The situation on the Chad-Sudan
border has become dangerous
Liria-Franch said humanitarian groups were also finding it harder to work with the local authorities as some of them have begun supporting the rebels.
The Sudanese government has denied Chad's charges that it is supporting and arming rebels from the United Front for Change, FUC.
But Deby has broken off diplomatic relations with Khartoum in retaliation for last week's uprising and in a stinging attack on Saturday called Omar al-Bashir, the president of Sudan, a "traitor".
Al-Bashir denied the accusation and blamed N'Djamena for instability on the border.