James Miller was killed on 3 May 2003 in the Rafah refugee camp while making the documentary Death in Gaza, about Palestinian children caught up in fighting against Israeli occupation.
Witnesses said Israeli soldiers shot him at close range, although he wore journalist insignia and waved a white flag.
The documentary shows Miller, 34, approaching an armoured vehicle in the dark before the fatal shots sound.
The army's top legal authority, Judge Advocate-General Avichai Mandelblith, met Miller's relatives and told them a two-year internal investigation had not uncovered proof of wrongdoing.
"The findings of the military police show that ... the commanding officer of the unit at the site allegedly fired his weapon in breach of Israel Defence Force rules of engagement," an army statement quoted Mandelblith as saying on Wednesday.
"However, it is not legally possible to link this shooting to the gun shot sustained by Mr Miller to provide a reasonable chance for conviction."
Miller's family cried foul.
"We do not believe there has been a genuine attempt to bring about justice," his sister, Anne, said.
She said the family would sue the army for compensation through Israeli civil courts. The suit would target the officer who fired the shots, once the army divulged his name, she said.
Human rights groups have said
soldiers use excessive force
Human rights groups have regularly accused Israeli troops of excessive force in fighting a Palestinian uprising that erupted in occupied Gaza and the West Bank in September 2000.
Hours after Miller's death, a senior Israeli commander said troops in Rafah had opened fire after coming under attack by Palestinian fighters who were smuggling in arms from nearby Egypt.
"Two of our armoured vehicles went in to uncover a smuggling tunnel ... and as darkness fell came under small arms fire. Two rocket-propelled grenades were also fired at them. The men shot back at the attackers," Colonel Avi, then the acting commander of Israeli forces in the Gaza Strip, had said.
Footage disputes account
Yet the footage in Death in Gaza shows no sign of a clash, and Avi's account was absent from the Mandelblith statement.
The judge advocate-general was quoted as saying the officer who fired his weapon would be "disciplined for misuse of his firearm" - but did not elaborate.
"Shockingly, this evidence has been destroyed, sabotaged or, at best, ignored by the IDF," Anne Miller said.
British authorities, including police coroners, have backed the Millers' bid to win an independent investigation of the killing.
"The British government will continue to raise James's case with the government of Israel," Foreign Office Minister Baroness Symons said in a statement.