Under the terms called for by Hamas, a deal would see Shalit released in exchange for 1,000 Palestinian prisoners held in Israeli jails.
Al Jazeera's Ayman Mohyeldin, reporting from the Gaza Strip, said Hamas was standing firm on its demands.
"The position of Hamas has been re-enforced to us this morning. They will not compromise on the list of demands which includes a full list of prisoners ... They will not accept anything other than that," he said.
"From their perspective, the ball is in Israel's court."
Netanyahu is said to be facing pressure from the families of Israelis killed by Palestinian fighters now being held in Israeli jail not to agree to their release.
According to Israeli media, the Israeli prime minister favours a plan that would see the prisoners sent instead to the Gaza Strip or into exile in foreign countries.
Al Jazeera's Sherine Tadros, reporting from Jerusalem, said the Israeli cabinet had held five rounds of often tense talks over the past 48 hours, in what had been billed as "make or break" time for the deal.
But instead the cabinet released a statement early on Tuesday instructing the team negotiating Shalit's release to continue their efforts to secure his return.
The statement essentially means that no decision was reached, Tadros said.
Daniel Levy, a former Israeli peace negotiator and now head of a US-based think-tank, said it was unclear whether the apparent deadlock in the Israeli cabinet was a case of brinkmanship, a rejection of the deal, or a calculated decision to postpone and drag out making a decision.
|Shalit was seized in a cross-border raid by Palestinian fighters in 2006 [AFP]
Speaking to Al Jazeera from New York, he said the Israeli public would most likely support a deal, "but the real question is whether Mr [Binyamin] Netanyahu [Israel's prime minister] is up for making a decision".
The issue of Shalit's detention had become a significant factor in the Israeli psyche, Levy said, reviving what he called the "national trauma" over Ron Arad, the now apparently deceased Israeli pilot captured during the Lebanon war.
If approved, the exchange would be subject to a 48-hour period for opponents to file legal challenges.
There has been increasing anticipation of a deal being reached in recent weeks.
Intermittent negotiations between Israel and Hamas have been conducted indirectly, mainly through Egypt, since Shalit was seized by fighters from Hamas and allied groups.
Germany, which has a history of organising successful prisoner exchange between Israel and Lebanon's Hezbollah, has also been involved in mediating in recent months.
The Associated Press news agency quoted a Palestinian source as saying that the Palestinian prisoners would be released in two stages.
A first group of 450 are to be freed as Shalit is handed over to the Egyptians, and then returned to Israel. The remainder would be released weeks or even months later.
More than 10,000 Palestinians are currently being held in Israeli jails.
In October, Israel freed about 20 Palestinian women prisoners in exchange for a video showing Shalit in captivity.
The video showed Shalit alive and moving - the first proof of his well-being since he was captured.