Azzam al-Ahmed, a Fatah politician, and Mousa Abu Marzouk, the Hamas chief negotiator, signed the deal in Saleh's presence at the Yemeni presidential headquarters.
Critics say the deal appears to bear little substance - agreeing only to open talks on a number of issues - and the two sides have offered differing interpretations of the signed document.
"Hamas and Fatah have agreed to accept the Yemeni initiative as a framework for dialogue ... and a return of the Palestinian situation to what it was before the events in Gaza [last June]," the declaration, read to journalists by Abu Bakr al-Qurbi, the Yemeni foreign minister, said.
He said the initiative would also "reconfirm the unity of the Palestinian homeland in terms of its land, people and the Palestinian Authority".
Yemen's peace initiative proposes seven points for Palestinian reconciliation
• Gaza must be returned to how it was prior to the Hamas takeover last June
• Agreement to hold early elections
• Resumption of dialogue on the basis of the 2005 Cairo agreement and the Mecca agreement of 2007
• Respecting the Palestinian Law and Basic Law and adherence to it by all parties
• Reconstruction of the Palestinian security institutions
• All Palestinian institutions to be free of any factional discrimination, subject to the law and the executive authorities
The Fatah-led Palestinian Authority insists that implementation of the proposal by Hamas is not open for negotiation.
"Resumption of dialogue [with Hamas] in the future must be to implement the Yemeni initiative in all its items and not to deal with the initiative as a framework for dialogue, because this will not yield an outcome," it said in a statement issued after the signing.
"The items of the Yemeni initiative are clear and we want it for implementation, not dialogue," Nabil Abu Rudeineh, an advisor to Mahmoud Abbas, the Palestinian president, said.
Hamas, on the other hand, says Sunday's agreement is a guideline for reconciliation talks, not a pre-condition.
Abu Marzouk told Al Jazeera: "There are two things [that are] very important - that we are going to return to a normal situation in Gaza and the West Bank, [and] the second one, that we consider the sticking points in the Yemeni initiative as the agenda of the negotiations."
The deal calls for the restructuring of Palestinian security institutions, with the armed forces intended to come under the control of the Palestinian Authority rather than holding loyalty to any faction.
The two sides also agreed to adhere to previous commitments, including those made by Palestinian factions in Cairo in 2005 and the Saudi-sponsored Mecca accord between Hamas and Fatah that led to the formation of a short-lived Palestinian unity government earlier last year.
One of the sticking points of the deal, however, is a return to the status quo before June 13 in both the Palestinian territories.
Fatah says the agreement stipulates that Hamas must return control of Gaza to the Palestinian Authority, but has so far shown unwillingness to make changes in the West Bank.
Azzam Ahmad, a Fatah spokesman, said: "There is no comparison between the situation in the West Bank and the Gaza Strip. The trouble started when Hamas took over in Gaza."
Hamas says rolling back all changes since June means the unity government, which it led - and Abbas subsequently dismissed in June - should be reinstated before it eases its grip on the Strip.
Ahmed Youssef, an adviser to Hamas leader Ismail Haniya, told Al Jazeera: "They say we are committing mistakes here [in Gaza] but they committed mistakes also in the West Bank."
Youssef said implementation of the agreement should address issues in the West Bank, such as a the violation of authority by Abbas in appointing an emergency government, as well as a crackdown on Hamas officials and the closing of affiliated political institutions in the territory.
However, the Hamas advisor eschewed optimism towards the agreement, saying: "We have laid the cornerstone never the less."