Iran had threatened before the five-hour meeting on Friday to resume sensitive atomic activities unless France, Britain and Germany agreed to allow it to carry out small-scale uranium enrichment.
"The informal talks have concluded. No conclusions were reached and both sides, the EU three and Iran, have agreed to go away and reflect on what was discussed and to continue the discussions in future," said a British Foreign Office spokesman.
But an Iranian official said some progress was made in the London talks.
Iranian negotiator Sirus Naseri told the AFP on Saturday: "We believe there has been some progress on the framework for a long-term agreement. The difference is about the timing."
Naseri was referring to a proposal concerning enrichment which the two sides discussed at a meeting on Friday.
Iran has been accused by the US of having a secret agenda to develop nuclear weapons. Tehran denies the charge, saying its nuclear programme is solely for the generation of power.
The EU's three biggest powers, which share Washington's concerns, are leading talks intended to persuade Iran to scrap its atomic fuel programme in exchange for economic and political incentives. Tehran has refused.
The latest proposal from Tehran suggests it be allowed to build up its uranium enrichment programme in stages, beginning with a small pilot enrichment plant and ending with a commercial-scale complex.
The EU powers hope to leave the hard negotiations on Tehran's atomic ambitions until after Iran's 17 June presidential elections, on grounds that campaigning for the ballot could produce heightened tensions.
"We don't want to break things up now and have a row. We want to continue the negotiating process after the Iranian election," said a European diplomat, declining to be identified.
The US wants Iran to scrap its
But Naseri said before the London meeting that he wanted agreement soon.
"The foundation for agreement is in place," said Naseri.
"We think it is unreasonable to avoid agreement," he added, insisting he was not putting undue pressure on the EU powers.
Iran has suspended its enrichment programme under international pressure, but four months of talks with the Europeans have yielded no breakthrough and Iran says the programme must resume.
"If there is no agreement and the Europeans insist on further time ... we may have to readjust the situation so it will be a more balanced position. It will not be balanced if the suspension will remain," said Naseri, in an apparent threat to resume enrichment unilaterally.
He later told Iran's official Islamic Republic News Agency (Irna): "In case of not reaching an agreement in London, Iran might be obliged to resume part of its uranium enrichment programme, but in that case it will still continue the talks."
'At any cost'
Akbar Hashemi Rafsanjani, an influential former Iranian president preparing to run again in June, said Tehran was determined to embark on uranium enrichment and other branches of nuclear technology.
"And we will have it at any cost," he told worshippers in Tehran.
Rafsanjani is a strong supporter
of uranium enrichment in Iran
Washington warned Tehran not to leave the negotiating table or resume any parts of its enrichment programme.
It also reminded the EU of its pledge to help refer Iran's case to the UN Security Council, which could lead to economic sanctions, if Tehran followed through on its latest threats.
"If Iran chooses to walk away from talks with the EU three and end its current suspension ... the EU three have already made clear to Iran that they would work with us and others to report Iran's nuclear programme to the UN Security Council," a US official in Vienna said.
EU diplomats say Tehran knows the idea of pilot enrichment is unacceptable to them and to Washington, which takes a harder line than the Europeans despite last month giving its backing to the diplomatic initiative.