He said the conference, which is expected to be held some time in the next two months, would not result in a final peace deal with the Palestinians and might not even take place at all.
"If all goes well, hopefully, we will meet in Annapolis," said Olmert on Thursday, "[But] Annapolis is not made to be the event for the declaration of peace."
Power cuts planned
Violence on Friday saw three Palestinian fighters killed by Israeli forces.
Israel has also recently been moving towards imposing a series of power cuts on the Gaza Strip in retaliation for rocket attacks fired from the territory.
Earlier, Ehud Barak, the defence minister, gave his final approval to cut off electricity to Gaza for longer periods each time rockets are fired into Israel.
Israeli officials would not say when the campaign would begin. Saeb Erekat, a negotiator for the West Bank-based Fatah administration, appealed for international intervention.
He called the decision to cut off power "particularly provocative given that Palestinians and Israelis are meeting to negotiate an agreement on the core issues for ending the conflict between them."
The US has been pushing the two sides to present a joint statement at the upcoming conference, paving the way for peace talks.
Palestinian officials said they hoped Abbas and Olmert would be able to overcome their differences over the content of the statement.
The Palestinians are seeking an agreement addressing the core issues at the heart of the conflict: final borders, the status of disputed Jerusalem and the fate of Palestinian refugees.
They also want a timeline for creation of a Palestinian state. But Israel wants a more general document, saying it is to soon for all of these issues to be addressed.
In Gaza, Ismail Haniyeh, who heads the territories Hamas administration, said the meetings between Abbas and Olmert were meant to distract from Israeli attacks and sanctions on Gaza.
He said the proposed peace conference would offer nothing to the Palestinians.