After intense negotiations, the Shas party said it was unable to reach an agreement with Kadima over the status of Jerusalem and social welfare benefits.
Shas said it had not received satisfactory offers on its two main demands.
"All through the negotiations we insisted on two points - substantial help for the [financially] weak in Israeli society and the defence of Jerusalem," Shas said in a statement.
"If the status of Jerusalem is not strengthened and we give the impression that Israel's capital could be part of a deal, that could have repercussions for all future negotiations."
Jacky Rowland, Al Jazeera's correspondent reporting from Jerusalem, said: "In order for Livni to scrape together a coalition now, she would have to appeal to parties at opposite ends of the political spectrum.
"It is very difficult to see how she could appeal ... and satisfy both a far-left party and a far-right party's demands."
Shas, which has long billed itself as a party that represents Israel's poor, has been demanding increased government spending of about $270m on social welfare as a price for joining a Livni-led coalition.
|Shas member Eli Yishai, right is deputy prime minister and minister of industry [EPA]
On Thursday, Livni set an ultimatum for Kadima's potential coalition partners, saying that if a government was not formed by Sunday, Israel would face snap elections.
Livni has the support of the Labour party, giving her a total of 48 seats in the 120-member Knesset, but is still short of a majority to form a strong government.
Shas's membership would have taken that number to 60.
Winning the support of smaller factions, such as the Pensioners party, with seven Knesset members, and left-wing Meretz, with five, would give Livni a stronger mandate to pursue policies that include peacemaking with the Palestinians.
Without Shas, Livni could form a minority government and rely on precarious support from left-wing and Arab parties wary of a national election that opinion polls indicate Benjamin Netanyahu's right-wing Likud party would win.
Livni has been trying to forge political partnerships since she was elected leader of Kadima last month when Ehud Olmert resigned as prime minister amid corruption allegations.
The 28-day period to form an administration expired on October 20, but Shimon Peres, the president, granted a two-week extension to avoid a general election.
Olmert remains in office in a caretaker capacity until a new government is formed.