Philippe Douste-Blazy, the French foreign minister, said the US-hosted meeting on Monday did not reach agreement.
"We are still considering our work," he told reporters after the talks.
An official who asked not be named said that "the prospects for an agreement this week are not substantially good".
Condoleezza Rice, the US secretary of state, hosted counterparts from Russia, Britain, France, China and Germany for talks that were also attended by Javier Solana, the European Union's foreign policy chief.
While the immediate issue was Iran's nuclear programme, Sean McCormack, the US state department spokesman, said the discussions "were much broader", including terrorism and regional security.
"The prospects for an agreement this week are not substantially good"
Ministers refused to answer reporters' questions during a picture-taking session. But McCormack said the meeting went on longer than planned because "it's an indication that all the ministers take this issue seriously".
Earlier, officials said Russia and China continued to resist a UN Security Council resolution sponsored by Britain and France and backed by the US that would legally require Iran to halt uranium enrichment.
The meeting of the Security Council's five veto-wielding permanent members plus Germany and Solana came after Mahmoud Ahmadinejad, the Iranian president, wrote to the US president proposing "new ways" to resolve their differences.
It was the first letter from an Iranian head of state to a US president since Washington broke off relations after the 1979 Islamic revolution.
Sceptical US officials dismissed the 18-page letter as a diversionary tactic that did not address what it sees as a problem - Tehran's nuclear research.
But a European diplomat who works on the Iran issue but was not authorised to speak publicly called the letter "another tactical masterstroke that was deliberately timed to come out today [before the ministers' meeting] and has made administration officials very nervous".
Iran is accused of secretly
developing an atomic bomb
Washington and its allies suspect Iran of developing nuclear weapons. Tehran says its programme is to supply energy and is purely civilian.
Earlier on Monday, China made clear that any reference to possible sanctions or war should be eliminated from the UN resolution ordering Tehran to curb its nuclear programme.
Moscow and Beijing want a resolution but oppose invoking Chapter 7 of the UN Charter, which is used routinely in Security Council resolutions for peacekeeping missions and other legally-binding actions.
The United States, France and Britain insist on Chapter 7. It allows for sanctions and even war, but a separate resolution would be required to invoke further steps of that nature.