Frank-Walter Steinmeier, the German foreign minister, said on Friday after the talks in Vienna that "time is running out".
He met with Iran's top nuclear negotiator, Ali Larijani.
"If we want success we have to act now," Steinmeier said.
He said the talks "were carried out in a constructive atmosphere, but finally we were unable to reach agreement".
Friday's talks came in the run-up to a 6 March meeting of the UN nuclear watchdog, which is to make an assessment of Iran's nuclear programme that will be sent to the UN Security Council for possible sanctions.
There had appeared little room for compromise, with Tehran insisting on its right to enrich uranium and Europe and the US saying Iran must give up the process, which makes nuclear reactor fuel and also atom bomb material.
Philippe Douste-Blazy, French foreign minister, said after the talks at the German ambassador's residence in Vienna that "unfortunately, we could not obtain an agreement with Mr Larijani".
Iran's Ali Larijani had requested
the hurriedly-arranged meeting
Also present at the hurriedly arranged meeting, which had been requested by Larijani, was Javier Solana, the European Union foreign policy chief, and a British official representing Jack Straw, the foreign minister, who was ill.
The EU troika this week warned Tehran in writing that progress depended on Iran stopping uranium enrichment and cooperating with UN nuclear inspections.
According to a letter from the trio's foreign ministers and Solana, "anything short of this would result in a public disagreement, which would set back our shared objectives".
The US had expressed scepticism about the prospects of success of Friday's talks.
Adam Ereli, US State Department deputy spokesman, said in Washington that "we're under no illusions".
"The baseline is the same. Is Iran going to suspend enrichment activity? Is Iran going to return to the negotiations? Or is Iran going to continue, as we think they have, to stall and prevaricate and extend things in a meaningless way in order to avoid censure," Ereli said.
Last month Iran resumed small-
scale uranium enrichment
Larijani had arrived in Vienna after talks in Moscow that also produced no agreement on a Russian proposal to enrich uranium for Iran on Russian soil - thus giving Tehran the fuel it needs but not the technology.
Larijani did not speak to the media after the talks.
The International Atomic Energy Agency (IAEA) had on 4 February reported Iran to the UN Security Council but allowed a month for diplomacy before the world body decides what measures, if any, to take against Tehran.
Unlike the IAEA, the Security Council has enforcement powers and it could impose sanctions. But it is expected first to urge Iran to co-operate with the nuclear agency.
The IAEA has called on Iran to suspend uranium enrichment as a confidence-building measure.
The Islamic republic last month fed a 10-centrifuge research cascade at a facility in Natanz with the feedstock uranium gas, signalling it was pushing ahead with enrichment it says is essential for a civilian energy programme.