The impasse has caused the worst internal fighting in a decade as the Hamas-led Palestinian government of Ismail Haniya, the prime minister, became embroiled in a power struggle with Mahmoud Abbas, the president, belonging to the once-dominant Fatah faction.
Yassir Abd Rabbu, a senior Abbas aide, said on Tuesday "The differences on the core issues have remained ... in the light of tonight's talks it does not seem as if we are closer to an agreement."
The main obstacles to the formation of a unity government have been Hamas' refusal to participate in any administration that recognises Israel and renounce armed struggle against it.
Abbas held two meetings with Sheikh Hamad bin Jassem bin Jabr Al Thani, the Qatari foreign minister, on Monday night, the second of which spilled over into the early hours of Tuesday morning.
Sheikh Hamad held back-to-back talks with Abbas and Haniya in the Gaza Strip after meeting earlier in the day with Hamas leader Khalid Mishaal in Damascus.
After his meeting with Haniya, Sheikh Hamad said that recognition of Israel remains the main obstacle to an agreement.
"The main problem is in mutual recognition [by Israel and the Palestinians] and how to establish the two states," he told journalists.
According to Abd Rabbu, the talks failed because Hamas continued to refuse to recognise Israel and to accept previously signed agreements with it.
"We will continue the dialogue over these points but no agenda for a unity government can succeed unless these points are resolved," he said.
"This initiative is the last political effort that is being exerted and the opportunity must be seized because the alternative is to hold early elections"
Yassir Abd Rabbu,
Abbas said the Qatari mediation effort would continue, but Sheikh Hamad left the coastal territory early on Tuesday and it was not clear if or when he would return.
Abd Rabbu said that the failure of the Qatari mediation would probably lead to early Palestinian elections.
"This initiative is the last political effort that is being exerted and the opportunity must be seized because the alternative is to hold early elections," he said.
Palestinian politicians said the Qatari proposals included forming a government of so-called technocrats and convening a meeting between Abbas and Mishaal, whose Hamas movement won Palestinian elections in January.
Fifteen people have been killed in clashes between Hamas and Fatah since talks on a coalition government have failed, the worst internal violence in Gaza and the West Bank since the start of Palestinian self-rule in 1994.
Abbas aides said Meshaal provided Sheikh Hamad with a counter-proposal by Hamas, the details of which were not disclosed. "Abbas was furious and rejected the Hamas paper," a senior Abbas aide said.