The Israeli military said a naval patrol spotted three boats, suspected of carrying smugglers, quickly approaching Gaza from the direction of Egypt before dawn and opened fire when the craft ignored warnings to turn back.
A military statement said the navy has repeatedly foiled past arms smuggling attempts.
Just hours before the Palestinian clashes, Haniya, Hamas' prime minister designate, said a Palestinian unity government would be announced in the next few days.
The agreement reached in the Saudi city of Mecca on February 8 had halted weeks of bloody factional fighting in which more than 90 people were killed in the Gaza Strip.
"The Palestinian people are in desperate need of leaders to solve their daily problems and security"
Send us your views
The man killed in Sunday's shootout was identified as Mohammad al-Kafarna, a member of the Hamas-led government's Executive Force.
Hamas accused Fatah of ambushing his car. Abdel-Halim Awad, a Fatah spokesman, said Hamas fighters had fired first on a car carrying members of the al-Aqsa Martyrs Brigades, a group which is linked to the Fatah faction.
Fighting quickly spread in Beit Hanoun as Hamas forces attacked a Fatah office and a separate security complex.
Later on Sunday witnesses heard gunfire in Gaza City, and a number of injuries were reported.
Fatah later denied any involvement in the shooting of al-Kafarna and a statement from Hamas only said he was killed by "suspect fire".
Hamas vowed revenge for Kafarna's death, saying "the blood of our leader will not go in vain".
The group said in a statement: "The criminals who planned and carried out this crime will regret it and they will not escape the tough and just punishment that they deserve."
Fatah security forces declared a high alert in northern Gaza, ordering their men to reinforce their positions and set up checkpoints to intercept Hamas vehicles.
Tension had risen on Saturday after Fatah gunmen shot at the convoy of a Hamas cabinet minister in the occupied West Bank, and armed men stormed the al-Quds University in Gaza City, wounding a Fatah student council member.