Police dug up the hospital's backyard and found pieces of bones, some wrapped in plastic bags.
Saxena said police conducted the search on Sunday after a tip-off about illegal abortions being carried out at the hospital and the subsequent dumping of foetuses near it.
"The question of female foeticide and infanticide is part of our investigation, as is illegal abortions," said Saxena.
Officials have said the practice of aborting or killing female children is spreading in India, a practice that some international groups say has killed more than 10 million female foetuses in the last two decades and led to an imbalance in the ratio between males and females in India.
Last December, the government said 10 million girls had been killed by their parents in the past 20 years either before they were born or immediately after.
Prenatal sex-determination tests are outlawed and the government has said it is clamping down on doctors who flout the law.
But social activists say there are a number of loopholes which allow those who provide tests to remain free and since the law was enacted, only one doctor has been convicted of illegally aborting female fetuses.
Despite the laws, the killing of female foetuses is still common in India and infanticide is also practiced in some areas.
The government's orphanage project, dubbed the "cradle scheme", is a bid to halt the abortions and killings.
Renuka Chowdhury, the minister of state for women and child development, has told an Indian news agency that the government planned to set up a cradle centres in each regional district.
"What we are saying to the people is have your children, don't kill them. And if you don't want a girl child, leave her to us," she said.
Chowdhury said she hoped that the cradle centres would also provide an opportunity for parents who had a change of heart to reclaim their children.
According to the latest census figures in India, the number of girls per 1,000 boys declined from 945 to 927 between 1991 and 2001 and many districts in the country routinely report only 800 girls born for every 1,000 boys.