In a statement on Wednesday, Human Rights Watch said the Israeli army (IDF) had excluded all evidence gathered by other sources.
It had either called into question or declined to accept evidence collected by the group, the statement said.
Marc Garlasco, a senior military analyst at Human Rights Watch, said: "An investigation that refuses to look at contradictory evidence can hardly be considered credible.
"The IDF's partisan approach highlights the need for an independent, international investigation."
Israel has ruled out an international inquiry.
The deaths on June 9, a day of heavy Israeli shelling, drew international condemnation and prompted Hamas, the ruling Palestinian movement, to call off its 16-month-old truce.
The army had no immediate comment on the accusations from the US-based human rights group, which has carried out its own investigation into the explosion that killed seven family members on an outing to the beach.
The army has said shelling of the area, in response to rocket fire into Israel, had ended before the beach blast.
Retrieved shrapnel samples also ruled out the possibility of a direct Israeli artillery barrage, it said.
But evidence collected by Human Rights Watch researchers indicated the civilians were killed within the time period of the shelling, the statement said.
"An investigation that refuses to look at contradictory evidence can hardly be considered credible"
Marc Garlasco, senior military analyst, Human Rights Watch
That evidence included computerised and hand-written hospital records showing the time when some of the wounded were admitted.
Offered such evidence in a meeting this week, the army dismissed the material as "unimportant", the statement added.
The Human Rights Watch statement follows a report by Israeli television on Tuesday which said the delayed explosion of a dud Israeli shell may have killed the beachgoers.
The army has not ruled this out.