In one case, the group said, Israeli agents convinced a suspect that his wife had also been arrested and tortured, driving him to attempt suicide.
In another, they detained a couple for an extended period, tortured them physically, and withheld information about their two young children to try to break them.
Yoav Loeff, a spokesman for the human rights group, said the Shin Beth is using relatives as leverage during interrogations.
"They use family members to force people to confess and they cross all the red lines along the way," he said.
The Shin Beth internal security agency, which conducts the interrogations, denied the charges.
It said in a statement that it never detains relatives of suspects, or presents false information to its suspects in an attempt to elicit information from them.
"Terrorist investigations are conducted by the Shin Beth according to the Supreme Court ruling, under the restrictions of the law and the tight supervision of the justice ministry and the courts," a statement read.
"The information acquired in these investigations allows for foiling acts of terror, and many civilians in Israel owe their lives to these actions."
A parliamentary committee heard the rights group's findings in a special session on Sunday.
A 1999 Israeli Supreme Court ruling severely limited the Shin Beth's interrogation methods, essentially outlawing torture except when there was clear evidence the suspect had information that could save people from an attack in the making.
The court outlawed what the Shin Beth called "moderate physical pressure", such as exposure to extreme temperatures and tying up detainees in painful positions.
But the Shin Beth has come under criticism for its new methods, which include shackling suspects in contorted positions and depriving them of sleep for long periods of time.