The rare finding could make it easier to discover good treatments for the disorder, one of the top 10 leading causes of disability worldwide.
Dr Norio Ozaki of Fujita Health University School of Medicine in Toyoake, Japan, and colleagues at several US institutions - including the University of Pittsburgh and Yale University - worked on the study, published in the journal Molecular Psychiatry.
The gene is called the human serotonin transporter gene, hSERT, and helps control how the body uses serotonin, a message-carrying chemical or neurotransmitter linked with mood.
Some anxiety drugs and antidepressants target serotonin, but the researchers said patients with the mutations are not helped by these drugs.
"In all of molecular medicine, there are few known instances where two variants within one gene have been found to alter the expression and regulation of the gene in a way that appears associated with symptoms of a disorder," said Dr. Dennis Murphy of the National Institute of Mental Health, who worked on the study.
"In all of molecular medicine, there are few known instances where two variants within one gene have been found to alter the expression and regulation of the gene"
Dr. Dennis Murphy
National Institute of Mental Health
The researchers analysed DNA from 170 people, including 30 patients with obsessive-compulsive disorder (OCD), 30 with eating disorders such as anorexia and 30 with seasonal affective disorder - which can cause depression and other symptoms in dark winter months.
They also looked at the DNA of 80 healthy people.
A specific mutation in the hSERT gene was seen in two patients with OCD and their families, but not in other patients.
With such a rare mutation showing up, the researchers believe it is likely to be found in other families with OCD and related disorders.
They interviewed relatives of the patients and found six of the seven people with the mutation had an obsessive-compulsive disorder, and some also had anorexia, Asperger's syndrome, which is a form of autism, social phobia or were abusers of
A second mutation was found in hSERT in two patients, giving them a "double dose." The patients and their siblings had especially difficult to treat versions of OCD, the researchers said.