"Neither candidate was able to enjoy the two-thirds majority," she said.
Feroukhi said that member states would have 28 days to nominate new candidates, and that Amano and Minty would be able to run for the post again.
ElBaradei, 66, will retire from office in November after three terms stretching over 12 years.
IAEA officials plan to secure his successor by June, to ensure a smooth transition.
The new director-general will have to face a number of challenges including controversy over Iran's nuclear programme and allegations that Syria has nuclear ambitions.
The change of guard comes at a time when the agency is seeking a significant increase in funding from member states over the next two years in order to carry out its duties effectively.
Steve Clemons, from the New America Foundation, told Al Jazeera that there was tension between those who fear Amano being a US puppet and desires from developing nations potentially with nuclear ambitions, who are seeing a pattern of nations being denied such technology.
"There's a real standoff between those like as Amano from Japan who want to be a classic preventer of non-proliferation. And those ... [like] Samad Minty ... who want to work with the developing world to provide legal mechanisms for the exchange of nuclear technology and for peaceful uses," Clemons said.
"And there's a real divide today because of the confusion about our nuclear non-proliferation treaty and where that's going to go."
Clemons said that other candidates may now enter the running for the position, and that there was a need for a director who could bridge the divide between an innovative outlook towards nuclear technology and its exchange and also prevention of proliferation.