Shalit was captured in a cross-border raid by Palestinian fighters in June 2006 and has been held by Hamas ever since.
Al Jazeera's Ayman Mohyeldin, reporting from the Gaza Strip, said Hamas was standing firm on its demands.
"The position of Hamas has been re-enforced to us this morning. They will not compromise on the list of demands which includes a full list of prisoners ... They will not accept anything other than that," he said.
"From their perspective, the ball is in Israel's court."
Binyamin Netanyahu, the Israeli prime minister, is said to be facing pressure from the families of Israelis killed by Palestinian fighters now being held in Israeli jails not to agree to their release.
According to Israeli media, Netanyahu favours a plan that would see the prisoners sent to the Gaza Strip or into exile in foreign countries.
If approved, the exchange would be subject to a 48-hour period for opponents to file legal challenges.
Al Jazeera's Sherine Tadros, reporting from Jerusalem, said the Israeli cabinet had held five rounds of often tense talks over the past 48 hours, in what had been billed as "make or break" time for the deal.
But instead the cabinet released a statement early on Tuesday instructing the team negotiating Shalit's release to continue their efforts to secure his return.
The statement essentially means that no decision was reached, Tadros said.
Daniel Levy, a former Israeli peace negotiator and now head of a US-based think-tank, said it was unclear whether the apparent deadlock in the Israeli cabinet was a case of brinkmanship, a rejection of the deal, or a calculated decision to postpone and drag out making a decision.
|Shalit was seized in a cross-border raid by Palestinian fighters in 2006 [AFP]
Speaking to Al Jazeera from New York, he said the Israeli public would most likely support a deal, "but the real question is whether Mr Netanyahu is up for making a decision".
The issue of Shalit's detention had become a significant factor in the Israeli psyche, Levy said, reviving what he called the "national trauma" over Ron Arad, the now apparently deceased Israeli pilot captured during the Lebanon war.
There has been increasing anticipation of a deal being reached in recent weeks.
Intermittent negotiations between Israel and Hamas have been conducted indirectly, mainly through Egypt, since Shalit was seized by fighters from Hamas and allied groups.
Germany, which has a history of organising successful prisoner exchange between Israel and Lebanon's Hezbollah, has also been involved in mediating in recent months.
The Associated Press news agency quoted a Palestinian source as saying that the Palestinian prisoners would be released in two stages if the deal were to go ahead.
A first group of 450 are to be freed as Shalit is handed over to the Egyptians, and then returned to Israel. The remainder would be released weeks or even months later.
More than 10,000 Palestinians are currently being held in Israeli jails.
In October, Israel freed about 20 Palestinian women prisoners in exchange for a video showing Shalit in captivity.
The video showed Shalit alive and moving - the first proof of his well-being since he was captured.