Solana said the meeting in London had been "a disaster".
The US, Britain, China, France, Russia and Germany agreed in September to delay sanctions against Iran until the end of November, pending reports on an investigation by the UN nuclear watchdog and a EU mediation effort.
They had decided that if the reports by the International Atomic Energy Agency (IAEA) and Solana did not show "a positive outcome", they would agree on more sanctions against Iran and put it to a vote in the security council.
The IAEA report found Iran was co-operating, but not proactively, making it likely that Britain, the United States and France will resume their push for sanctions.
In previous meetings, Russia and China, which have strong trade ties with Iran, have agreed only to the mildest of measures.
But a diplomatic source told the AFP news agency that the five UN Security Council members, along with Germany, had agreed to examine "the elements of a new resolution on sanctions".
"A compromise text will be worked out and should circulate between the capital cities concerned next week," the source said.
Tehran had promised to bring "new ideas" to the table for the London talks but Cristina Gallach, Solana's spokeswoman, said: "There was not enough new in order not to be disappointed."
However, Jalili said on Saturday that Iran was not to blame for the disappointment expressed by Solana.
"The fact is that we defended the Iranian nation's rights and stressed fulfiling our duties and that the Iranian nation will not accept anything that goes beyond the nuclear Non-Proliferation Treaty," he said.
"If some people have become disappointed because they cannot deprive Iran of its natural rights then this is another matter," he told reporters after arriving back in Tehran.
Tehran rejects charges it is pursuing nuclear weapons under the cover of a civilian atomic programme.
It says it only wants to generate electricity, but its failure to allay international fears has prompted two rounds of UN sanctions.