"The material [B'Tselem] has collected, including an analysis of the area, photographs of bodies, and eyewitness accounts, raise doubt about the IDF spokesperson's contention that a secondary explosion is what killed the family," the group said.
The military released a video of the strike in which four suspected Palestinian fighters, armed with weapons, are seen walking near a home. However, it was unclear from the footage what killed the family.
David Chater, Al Jazeera's correspondent in Jerusalem, said that the "real time sequence and the claims by the Israeli army have not been subjected to independent expert analysis".
Their interpretation is based solely on the video footage and is at odds with field researchers from B'Tselem who gathered evidence on the ground immediately after the strike.
"They [B'Tselem] say it is not enough to launch an internal inquiry alone."
The Israeli military said that the air force twice fired missiles at Palestinian fighters "carrying backpacks loaded with ammunition" near the home.
"One gunman was targeted and hit from the air. As a result a strong secondary explosion occurred," an Israeli military statement said.
"The second gunman was targeted and hit as well, causing an even bigger explosion ... Both explosions were significantly stronger than those caused by the IDF [Israeli Defence Force] attacks against them."
A 17-year-old Palestinian student was also killed in the same incident, according to hospital officials.
Ibrahim Abu Meatak, a close relative of the victims, dismissed the Israeli military's report as a lie.
|Gazan were outraged by the killings [AFP]|
"We knew they were not going to treat us fairly. Other families have been eliminated before and they didn't take responsibility," he told the Reuters news agency.
B'tselem had called for a criminal investigation into the killings, saying the army appeared to have violated international law by firing close to the family's home.
The group said their investigation found that the family were killed by a missile fired at a Palestinian fighter who was about one metre from their home.
The shockwave from the blast tore off an iron gate that flew into the home, killing the family while they ate breakfast, according to a letter sent to Israeli military officials by the group.
The Israeli military said the operation in Beit Hanoun was launched to "ensure that rocket crews, snipers and tunnel diggers are kept away from the border".
Israel says the fighters endangered the lives of the family by operating near their home.
"They were going out to battle in the middle of a residential neighborhood," Major Avital Leibovitz, an army spokeswoman, said on Friday.
However, Hamas and the family's neighbours denied that Palestinian fighters were operating near the home during the Israeli attacks.
The killing of Abu Megteg and her four children - whose ages ranged from one-to-five years-old - dealt a blow to Egyptian efforts to broker a Gaza truce between Hamas and Israel.
Hamas deplored the deaths as a "war crime" and reportedly fired rockets into Israel following the killings.
Ehud Olmert, the Israeli prime minister, said he was "deeply sorry" about the deaths but blamed Hamas for operating among civilians.