An Israeli army spokesman told the AFP news agency that the Gaza incursion was launched against Palestinian fighters who were firing Qassam rockets near the town of Bait Lahiya.
Fourteen others were wounded in the shootout on Monday, the first day of the three-day Muslim Eid al-Fitr holiday marking the end of the Ramadan fasting month, Palestinian hospital officials said.
The Israeli army said armed men fired on its troops, who were in the area operating against Palestinian rocket-launchers. Ten Palestinians were hit in the ensuing gun battle, it said.
The Popular Resistance Committee, an armed Palestinian group, said that the operation was aimed at Ata Shindari, who it said organised the rocket attacks, as he visited a mourning tent for a man shot dead by Israeli soldiers in an earlier incident.
Residents said that Shindari and at least one of his brothers were killed in the Israeli attack, along with several more of his relatives.
Palestinian fighters in the tent initially returned fire at the house of a local family embroiled in a feud with the mourners, believing them responsible, before realising their mistake and shooting at the Israeli soldiers, residents told AFP.
Israel had tried four times in the past to kill Shindari, a witness said.
Aljazeera also reported that a Palestinian has been killed in an Israeli raid in the West Bank.
Mohammed Beni Auda, 23, was shot in the head, when Israeli troops raided the village of Tamoun in the West Bank, it was reported.
The incursions came as Ismail Haniya, the Palestinian prime minister, appealed for Palestinian unity and for an end to internal feuding between his ruling Hamas party and the rival Fatah faction led by Mahmoud Abbas, the Palestinian president.
Both sides have clashed repeatedly since the Islamist Hamas movement defeated Fatah in January elections and at least 19 people have died as violence escalated in the past few weeks.
"Stop the bloodshed, stop using weapons against your brothers and unite"
Palestinian Prime Minister
"Stop the bloodshed, stop using weapons against your brothers and unite," Haniya told a crowd of about 20,000 gathered in Gaza City following prayers for the Eid al-Fitr feast marking the end of Ramadan.
On Sunday, a senior member of the Palestinian security services loyal to Abbas's Fatah movement was shot dead at the door of his home near the Buraij refugee camp in Gaza after protests by Palestinian security personnel over unpaid salaries turned violent.
Most of the security forces are loyal to Abbas, but since the January elections Hamas has formed its own security force of 6,000 men and deployed it throughout the Gaza Strip.