The lack of a majority government has put the Shas party in a strong position to press for its policies - increased family benefits that would cost $400m, and a promise not to negotiate with the Palestinians over the future status of Jerusalem.
On Monday, Shimon Peres, the president, had given Livni a two-week extension to the original 28-day period allowed to form her administration.
Livni, who is presently the foreign minister, was elected leader of the Kadima party last month after Ehud Olmert resigned as prime minister amid a corruption scandal.
Last week Kadima, which has 29 seats in the 120-member parliament, reached a draft coalition agreement with the Labour party, which has 19 MPs.
But major differences emerged in negotiations with Shas, which has 12 MPs and has played the role of kingmaker in the past.
The Pensioners party, which has seven parliamentary seats, is demanding a rise in pensions that could cost another $900m, and talks have been more difficult than expected.
'No more haggling'
Tzahi Hanegbi, a Kadima party Knesset member, who has been leading the negotiations, said his party had gone as far as it could.
"There is a limit to the haggling," he told Israeli public radio. "This is true for Shas, the Pensioners party and for the others who have failed to fully appreciate Tzipi Livni's determination.
"Enough of this bazaar."
If a new government is not formed snap polls will be held in 2009 with the possibility that the vote could return the right-wing Likud party to power.
As foreign minister, Livni has been leading US-backed negotiations with the Palestinians since the November 2007 Annapolis conference that revived peace talks.
She has said she would continue the peace process if she becomes prime minister.