Gilad Shalit was seized by Palestinian fighters in a cross-border raid in June 2006.
In an interview with Israeli newspaper Haaretz, Shalit accused Olmert of violating his commitments and of being insensitive towards captured soldiers.
"Olmert told us he does not have a contract that requires him to free any citizen from captivity," said Shalit, referring to one of his meetings with the prime minister.
"That is not acceptable to me, and I don't think it is acceptable to anyone in this country."
|Gilad Shalit, right, was captured by Hamas
fighters two years ago [EPA]
Shalit also urged the Israeli government to maintain the economic siege on the Gaza Strip until his son's fate was resolved.
Ehud Olmert, the Israeli prime minister, said on Wednesday the captured soldier's release was an inseparable part of the ceasefire.
But Hamas leaders say the soldier's release depended on Israel freeing hundreds of Palestinians prisoners.