Palestinian security sources witnessed about 40 tanks pulling out of the area on Saturday night, but although an Israeli military source confirmed the withdrawal, he hinted it was only temporary.
International condemnation of Israeli action followed after the full extent of the damage caused by the military invasion began to emerge.
Israel claimed the devastating attack was to destroy weapons-smuggling tunnels snaking under the border between the Gaza Strip and Egypt and would "continue as long as is necessary", a spokesman said.
Despite the pullback, six more Palestinians were injured by Israeli gunfire on Sunday, as troops fired intermittently from the Israeli-controlled border area to which they retreated, medical sources said.
"We announce Rafah is a city of disaster," said Rafah Governor, Majid al-Ghal, on Sunday, pointing out that at least 100 houses had been destroyed during the operation.
"They have destroyed the roads, the water supplies, sewage, telephones, electricity," he said.
A local source for the Palestinian Centre for Human Rights also said "at least 100 houses" had been destroyed.
Peter Hansen, head of the UN Relief and Works Agency (UNRWA), said about 1500 people had been left homeless as a result of the operation.
"We have had very, very significant damage to the refugee camp," he said.
Hamas fighters resisted the Israeli
"It would appear between 100 and 120 shelters/houses were completely destroyed or demolished ... if these initial estimates are correct, it will mean we will have some 1500 persons added to the homeless roll in Rafah," Hansen said.
Governor al-Ghal was deeply sceptical the aim of the Israeli operation was merely to destroy the tunnels.
"They want a buffer zone near the border. The subject of tunnels is a false accusation and a lie. Why destroy trees, agricultural land?"
Fighter shot dead
Eight Palestinians were killed, including two children and two teenagers, and about 80 injured during the operation launched on Thursday night.
Resisting the raid, the armed wing of Hamas said it fired a makeshift Qassam rocket towards the Israeli town of Sderot. The rocket caused no damage or injuries.
Meanwhile, a Palestinian fighter was shot dead late on Saturday by Israeli soldiers near the Jewish settlement of Morag in southern Gaza.
"The subject of tunnels is a false accusation and a lie. Why destroy trees, agricultural land?"
governor of Rafah
The leftist Popular Front for the Liberation of Palestine announced by loudspeaker in Rafah that one of its fighters had been killed, witnesses said.
Further north, two Palestinians were wounded by Israeli gunfire in Khan Yunis, Palestinian medical sources said. A four-year-old girl was said to be in critical condition after being hit in the neck by a bullet.
Palestinian security sources said Israeli troops began shooting after resistance fighters fired a mortar shell towards the Jewish settlement of Ganei Tal.
Israeli occupation troops were also active in the northern West Bank late on Sunday, with about 15 tanks and jeeps pushing into the town of Kafr Rai, just south of Jenin, witnesses said.
The forces converged on five houses in the town and arrested 10 people, security sources said.