The fire at the orphanage's baby ward was put out after 6am (04:00GMT).
Vladovic said the fire began in a room where babies were sleeping. The infants were evacuated and taken to hospital.
Vladovic said that it had not been a major fire and that the babies probably died from smoke inhalation.
A spokeswoman for the Sarajevo clinic said 23 children, including two babies who were already dead, had arrived at the hospital on Sunday morning. One of those in a critical state later died in the hospital.
Prosecutors said another two babies had died at the scene of the fire. Two more remain in a critical condition.
The Sarajevo orphanage is home to 110 children and all the babies sleep on the third floor where the fire broke out.
Police said they were still investigating the cause of the fire.