Three senior hospital officials were also sacked or suspended and the government ordered the Jining Municipal Health Bureau to make a public apology.
"It exposes a serious loophole in the hospital's management and indicates a lack of ethics and legal awareness of some hospital staff," Gong was quoted as saying.
"It exerts a very negative impact on society and teaches us a profound lesson."
The two detained workers reportedly said the families of the dead babies had paid them to dispose of the bodies, but they instead dumped the bodies in the river.
It is believed the babies died of natural causes.
"These were all at once stage living babies - some of them premature, others had other serious conditions which resulted them being in the intensive care unit," Al Jazeera's Harry Fawcett, reporting from Jining, said after talking to the deputy head of the hospital.
"[The deputy hospital chief] pretty much admitted that the hospital was greatly at fault in almost all aspects of this case.
"She said the management had failed, that there was a failure in the training of these two people involved, that the hospital regulations were perfectly adequate but that there was poor enforcement of those regulations in terms of making sure that those things could not happen."
Eight of the corpses had identity tags that helped investigators trace them back to the Affiliated Hospital of Jining Medical University, Xinhua reported.
Residents found the bodies of the babies at the weekend, stashed in yellow plastic bags washed ashore under a bridge along the Gaungfu river in Shandong province.
The website of the Shandong Broadcasting Company showed bodies, some uncovered and others in bags, lying on parts of the bank of the river.
The bodies of the babies were cremated on Tuesday night.
Thirteen of the bodies have not yet been identified. Video footage showed that at least one yellow plastic bag was marked "medical waste".
The number of girls or boys was not reported.