At least 16 Palestinians were wounded in the fighting, which pitched Palestinian fighters against Israeli troops backed by attack aircraft.
Hamas said its fighters fired rifles and rocket-propelled grenades at the Israeli troops in the largely residential area, while Israeli snipers fired at Palestinian fighters.
Nour Odeh, Al Jazeera's Gaza correspondent, said that up to 80 Palestinians have also been detained by the Israeli army.
An Israeli army spokesman had said the operation was carried out to try to find a suspected tunnel network and prevent cross-border Palestinian rocket fire.
The spokesman said: "During the operation, our forces encountered gunmen and fired on them, hitting at least four."
He did not say whether any tunnels were found during the operation.
The Izz el-Din al-Qassam Brigades, Hamas's armed wing, said in a statement that they would fight against Israel "until either victory or martyrdom".
Israel has tried to isolate Hamas in the Gaza Strip, which the group seized by force in June in fighting with Fatah, the secular faction led by Mahmoud Abbas, the Palestinian president.