The discovery could explain why babies lying face down are more likely to die.
"These findings provide evidence that SIDS [Sudden Infant Death Syndrome] is not a mystery but a disorder that we can investigate with scientific methods, and some day, may be able to identify and treat," Dr Hannah Kinney of the Boston hospital, an author of the paper, said.
The study, published in this week's Journal of the American Medical Association, was based on autopsy data from 31 infants who had died from SIDS and 10 who had died from other causes between 1997 and 2005 in California.
In the SIDS victims, a look at the lowest part of the brainstem, the medulla oblongata, found abnormalities in nerve cells that make and use serotonin, one of the chemicals in the brain that transmit messages between nerve cells.
When infants are face down their reflexes, including turning their heads, are harder to trigger when their breathing is challenged, and may explain why they are more likely to die in this position, the report from Children's Hospital Boston and Harvard Medical School said.
"These findings provide evidence that SIDS [sudden infant death syndrome] is not a mystery but a disorder that we can investigate with scientific methods, and some day, may be able to identify and treat."
Dr Hannah Kinney, an author of the report on Sudden Infant Death Syndrome
Serotonin and how it is processed in the brainstem may help co-ordinate breathing, blood pressure, sensitivity to carbon dioxide and temperature, the report said.
When babies sleep face down or have their faces covered by bedding, they are thought to breathe exhaled carbon dioxide back in, depriving them of oxygen.
"A normal baby will wake up, turn over, and start breathing faster when carbon dioxide levels rise," said Kinney.
But babies with the defect die because the reflexes are impaired, she said.
Since 1994 public health officials have advised parents to put infants to sleep on their back. The campaign has cut the number of SIDS deaths in the United States by half.
Despite this it is still the leading cause of death among babies between three weeks and five months of age, with a mortality rate of 67 per 100,000.
Reducing the risk
Parents have also been advised to avoid soft bedding, sharing a bed with an infant, smoking during pregnancy and smoking around a baby after birth to reduce the risk.
Among the SIDS infants in the study, 65 per cent were sleeping prone or on their side at the time of death - indicating a need for continued public information on the problem, the study said.
The findings may lead to the development of a test to identify infants that are at risk, or perhaps some day a drug or other type of treatment to protect infants with abnormal brainstem serotonin systems, the authors said.