The Yedioth Aronoth said on Friday that 63 percent of the respondents favoured Olmert stepping down.
The paper described the poll as a political "earthquake".
Olmert was not the only politician to fare badly, with 74 per cent of those surveyed saying that Amir Peretz, the defence minister, should resign and 54 percent wanting Dan Halutz, the head of the military, to go.
A career politician with little military experience, Olmert has faced criticism for the perceived lack of success of Israel's campaign against Hezbollah in Lebanon.
Boost for opposition
Friday's poll also indicated that if an election were held tomorrow the country's opposition, led by the Likud party that was heavily defeated in the general election in March, would win.
Any vote held now would see Likud, led by Benyamin Netanyahu, increasing its share of the 120-seat Knesset by 20 seats and the far right Russian-oriented Yisrael Beitenu going from 11 to 17 MPs.
However, only 27 per cent of Israelis want snap elections, with 50 percent saying they would prefer right-wing opposition parties brought into the current government.
Israelis have also retained their confidence in their army with the institution enjoying support levels of 82 per cent.
The survey, carried out by an independent institute, was based on a representative sample of 500 people and carried a 4.5 percent margin of error.