Carrying banners bearing messages such as "Yes to peace, no to violence", the crowd assembled at the same square where Rabin was assassinated, now renamed in his honour.
David Grossman, an Israeli writer whose son was killed during the summer war in Lebanon, addressed the crowd, calling for the government to re-start peace talks with Palestinians.
He said: "Time is not on Israel's side. Why do hundreds of Palestinians and our soldiers have to fall?"
He attacked what he called "racism toward the Arab Israeli minority [and] social indifference" saying Israel was "going through one of the worst moral crises of its history".
Yigal Amir, an ultra-nationalist, shot Rabin after a peace rally in Tel Aviv on November 4, 1995 for initiating peace negotiations with the Palestinians.
A recent poll found that one in three Israelis were willing to see Amir, who has never expressed regret for the murder, eventually pardoned.
Rabin inspired both admiration and hatred for signing the 1993 Oslo autonomy accords with the Palestinians.
He shared the 1994 Nobel peace prize with Shimon Peres, the then foreign minister and current deputy prime minister, and Yasser Arafat, the late Palestinian leader, for the agreement.