ElBaradei says IAEA is unable to verify some important aspects of nuclear programme.
Friday’s meeting was seen as a last chance to avert further US-led sanctions against Tehran.
Jalili said after the meeting that it was “unacceptable” to demand Iran halt its uranium enrichment programme.
“If some countries want to use the UN Security Council and its resolutions to stop Iran‘s atomic work, surely they will not be successful,” Jalili said.
The US and other world powers say Iran’s nuclear programme is aimed at building atomic weapons.
Iran says its uranium enrichment programme is only intended to produce electricity.
Attempts by Britain, China, France, Germany, Russia and the US to end Tehran’s nuclear ambitions have failed so far.
The six nations said they would pass a new UN Security Council resolution if there was no progress by December.
“It’s quite clear from Solana’s public remarks that the Iranians have not agreed to comply fully with the requirements of the Security Council to suspend its enrichment programme,” a spokesman for Britain’s foreign ministry said.
The five permanent members of the Security Council plus Germany plan to draft a new resolution imposing wider financial, trade and visa restrictions on Iran.
Mohammed Shakeel, an Iran analyst at the Economist Intelligence Unit, told Al Jazeera that differences remained over whether sanctions should be applied.
“[Sanctions] depend on what the Russians and the Chinese make of these latest round of talks,” he said.
“Iran is relying on those two countries, both veto-wielding powers within the Security Council. For now the Chinese are quite reluctant to impose anything more severe.”
Britain, France and the US are thought to be considering more serious restrictions on Iran than Russia, China and Germany, three nations with close commercial ties to Tehran.
Shakeel said Iran has insisted that current sanctions have not affected its nuclear programme.
“It’s very interesting to hear Iranian nuclear negotiators insisting that existing sanctions resolutions have not really done anything against Iran,” he said.
“They are targeting Iran‘s economy… but on the wider nuclear front Iran is adamant… that its nuclear programme is progressing.”
Tehran insists that it has been co-operative with the International Atomic Energy Agency (IAEA), but has prevented inspections beyond uranium production sites since the nuclear case was referred to the UN Security Council in February 2006.