Hamas said its fighters were looking out for Israeli military movements near the fenced area when they came under attack.
Six Hamas fighters were killed in an Israeli missile strike on Monday.
In one of Tuesday's attack, two Palestinian boys were killed and three others wounded when the Israeli army launched a missile.
A Palestinian health ministry official said the two boys - aged 11 and 12 - were in an area close to where fighters had fired rockets into Israel.
The Israeli army said it fired on two Palestinians standing next to a rocket launcher following a rocket attack by Hamas fighters.
Earlier on Tuesday, an Israeli air attack killed three fighters from Islamic Jihad in the Gaza Strip near the border area.
In a text message to journalists, the group said that the men were killed east of the town of Khan Yunis, as the result of a direct hit.
An Israeli army spokesman said that the military "attacked and identified hitting three armed gunmen that were identified close to the security fence in the central Gaza Strip".
Israel has carried out a number of operations inside Gaza since Hamas took full control of the territory in June.
Residents initially reported that a missile hit a car in the Khan Yunis attack, but later said that the fighters were on foot.
The three men were taken to the Nasir hospital mortuary in Khan Yunis.
An Islamic Jihad spokesman said the group would continue firing rockets into Israel.
West Bank raid
Earlier on Tuesday, a Palestinian fighter was shot dead in clashes with Israeli soldiers in the al-Ain refugee camp in the West Bank city of Nablus.
Nasir Mabrouk belonged to the Abu Ali Mustafa Brigades, the armed wing of the Popular Front for the Liberation of Palestine.
The Israeli operation in the camp, which started two days ago, has included the arrest of a number of people living there and the searching of many buildings.
On Monday, six Hamas fighters were killed in an Israeli air strike on a Jeep in the al-Bureij refugee camp in central Gaza.
The latest deaths took to 5,831 the number of people killed in Israeli-Palestinian violence since the start of the second Palestinian uprising in 2000, the vast majority of them Palestinian.