The crowd - numbered at 25,000 by the police and described as tens of thousands by organisers - held Israeli flags and banners demanding a state commission headed by a judge and urging Olmert, Amir Peretz, the defence minister, and Dan Halutz, the chief of staff, to resign.
The demonstration was organised by the Movement for Quality Government and reservists who have criticised the government for its handling of the war.
Eliad Shraga, the founder of the Movement for Quality Government, told the crowd: "We want a government that doesn't close its eyes, that takes responsibility for the whole situation."
Speakers from the right and left of the Israeli political spectrum addressed the crowd.
Yossi Sarid, the former education minister and a member of the left-wing opposition Meretz party, said: "Olmert, Peretz and Halutz are dead men walking."
Moshe Arens, the former defence minister and a member of the right-wing Likud party, said: "Never has there been such confusion and contradictory orders issued in the handling of a war."
It was the largest public show of dissatisfaction with Olmert since he took office in May. Polls show that his popularity has tumbled amid public frustration with Israel's failure to crush Hezbollah and the conduct of the 34-day war, in which some 162 Israelis were killed and some 4,000 rockets fired on northern Israel.
A group of army reservists have criticised military "indecisiveness", unclear war aims, confused orders, food, fuel and water shortages, and the slowness to launch a major ground assault.
But Olmert has claimed that the establishment of a state commission - the most powerful type of inquiry in Israel - would "completely paralyse" the leadership at a time of external threats.
Olmert has pledged only a government probe into the war, under a panel his cabinet has yet to name. Israeli media has said that the cabinet was likely to put off a decision on naming the panel that had been expected on Sunday.