Leaders of Hungary, the Czech Republic, Poland and Slovakia, known as the Visegrad group (V4) were to meet in Israel for two days beginning on Monday.
Poland’s withdrawal came after acting Israeli foreign minister Yisrael Katz said on Sunday that Poles "collaborated with the Nazis" and "sucked anti-Semitism with their mothers' milk".
They were the latest comments in a diplomatic row that has been escalating since Friday between Poland and Israel about atrocities that occurred against the Jews on Polish soil during World War II.
Israeli Prime Minister Benjamin Netanyahu reportedly said during his trip to Poland on Thursday that "Poles cooperated with the Germans" during the Holocaust.
Netanyahu later said he had been misquoted, and that he did not mean all Poles cooperated, just certain ones.
'Anti-Semitism was innate'
Poland's Prime Minister Mateusz Morawiecki announced on Sunday that he was pulling out of the meeting because of Netanyahu's comment and that his foreign minister would attend the meeting in his place.
Katz exacerbated the problem with remarks on Sunday in an interview on Israeli TV.
"Poles collaborated with the Nazis, definitely. Collaborated with the Nazis," he said.
"As (former Israeli Prime Minister) Yitzhak Shamir said - his father was murdered by Poles - he said that from his point of view they sucked anti-Semitism with their mothers' milk. You can't sugarcoat this history," he said.
Katz told Israel's Army Radio on Monday that he was in favour of maintaining relations with Poland, but he repeated his earlier opinions.
"Historical truth cannot be changed. Many Poles collaborated with the Nazis and took part in the destruction of the Jews during the Holocaust ... Anti-Semitism was innate among the Poles before the Holocaust, during it and after it, too," he said.
Morawiecki withdrew the entire Polish delegation on Monday.
"Not only can we not accept such racist comments, but with all our strength we want to stress that we will fight for historical truth, for the honour of Poles," he told reporters.
"It is really astonishing that the newly appointed foreign minister of Israel quotes such a shameful and racist remark. Utterly unacceptable," Poland's ambassador to Israel Marek Magierowski said on Twitter regarding Katz's comments.
Poland's Foreign Ministry also summoned the Israeli ambassador, Anna Azari, to demand a second set of clarifications in recent days.
The developments mark a new low in a bitter conflict between Poland and Israel over how to remember and characterise Polish actions towards Jews during the German occupation of Poland in World War II.
This is not the first time Israel and Poland have been at loggerheads over this issue.
Poland last year introduced new legislation making the use of such phrases as "Polish death camps" punishable by up to three years in prison. After pressure from the United States and an outcry in Israel, Poland watered down the legislation, scrapping the prison sentences.
"[This issue] has its own domestic dimensions. We are fragile in terms of any accusations or guilt regarding the Holocaust. Every time somebody mentions this we protest, so we feel this is unfair," said Olgierd Annusewicz, a political scientist at Warsaw University.
"Millions of Jews were killed during World War II, but at the same time millions of Poles were killed as well," he said.
While the V4 summit has been cancelled, Czech Prime Minister Andrej Babis of the Czech Republic, Slovak Prime Minister Peter Pellegrini and Hungarian Prime Minister Viktor Orban are already in Israel and would instead hold bilateral meetings with Netanyahu.
"We have more or less agreed that the V4 meeting could only be in the second half of this year when the Czech Republic is to take over the presidency," Babis told reporters.
"An important decision to postpone the meeting with Israel shows that the V4 is one and there is no agreement among us for baseless racist attacks on any of the partners," the Polish prime minister wrote on Monday on Twitter.
Netanyahu was looking to the V4 summit to help burnish his diplomatic credentials before Israeli elections on April 9.
The V4 nations have pro-Israeli governments that often counter criticism against Israel in international forums.
Philip Heijmans in Prague contributed to this report