Washington has put pressure on Israel to increase the number of prisoner releases, as well as to go beyond a partial freeze on the building of settlements in the West Bank.
"The question is will Fatah be strong enough to contain such a powerful 'partner' or will it be just exploited and eroded over time?
Send us your views
But Israeli officials said they would not be able to release many more prisoners without changing existing criteria, which exclude Hamas members and those who have "blood on their hands", a reference to deadly attacks against Israelis.
Olmert also told his cabinet on Monday that Israel would not build any new settlements in the West Bank.
But he stopped short of announcing a freeze on construction in existing communities.
He promised to remove unauthorised settlement outposts in the West Bank, but gave no timeline for taking action. Israel has made similar pledges in the past.
"Let's be straight, we committed ourselves in the road map not to build new settlements and we will not build any," Olmert was quoted as saying by Miri Eisin, his spokeswoman.
The Israeli leader was referring to the 2003 "road map" that calls on Israel to freeze all settlement activity and for the Palestinians to rein in fighters.
Jacky Rowland, Al Jazeera's correspondent in Jerusalem, said: "The Israelis will say there are no new settlements being built, but you will see whole areas of building taking place on hills in the West Bank."
"The Israelis will say 'this is not a new settlement, it is a new neighbourhood of an existing settlement'," she said. "They descirbe it as 'natural growth' of an existsing settlement."
"It's this kind of use of language and rather creative interpretation of promises made which, in the view of many critics ... renders these undertakings of the government meaningless."
At the start of the weekly cabinet session on Monday, Olmert announced he would leave for the Annapolis meeting "at the beginning of the week".
He said he hoped Israel and the Palestinians could still reach agreement "on the procedural side" of the conference.
The US has pressured the two sides to submit a joint statement addressing in general terms issues such as borders, and the future of Jerusalem and Palestinian refugees.
But disputes between Israel and the Palestinians have cast doubts as to whether the two sides will be able to prepare the document.
The Palestinian leader dispatched aides to Washington on Sunday to try to narrow differences on the joint document.
Palestinian officials have suggested reaching an agreement on the joint document was crucial to ensuring participation of key Arab states such as Saudi Arabia.
US and Israeli officials said the paper was not a precondition for the conference, which was called by George Bush, the US president.
Israel, the Palestinians and the US have said the centrepiece of the conference is to be an agreement to begin formal negotiations on a Palestinian state, with the goal of reaching a deal before Bush leaves office in January 2009.
Olmert told his cabinet the post-Annapolis negotiations would be "very intensive and serious and deal with all of the core issues that are part of the process that should lead to a two-state solution".
Arab League foreign ministers are to meet in Cairo on Friday to decide whether to attend the US-sponsored conference.
Olmert's office said the Israeli leader planned to go to Egypt on Tuesday for talks with Hosni Mubarak, the Egyptian president.
The trip appeared to be part of a diplomatic effort to ensure broad Arab participation in the Annapolis meeting.
The US-sponsored conference comes at a time when both Abbas and Olmert are weakened.
Olmert is facing police investigations over alleged corruption, which he has denied, and the results of an official inquiry into his handling of the 2006 Lebanon war.
Fatah, led by Abbas, was driven out of the Gaza Strip in June by its rival Hamas movement.