He said: "I deeply regret that some have chosen to stand aside. I hope they will not do so for long".
"Some nations who by rights should be helping us to forge a path to a better future are not here.
"Outside these halls, interest groups of many political and ideological stripes shout against one another in acrimony," Ban said.
Before he set off for Switzerland, Ahmadinejad was quoted by Iran's state broadcaster as saying "the Zionist ideology and regime are the flag bearers of racism".
The president is due to speak at the conference at 1300 GMT.
Israel withdraws envoy
Earlier on Monday, Israel withdrew its ambassador to Switzerland in protest over a meeting between Ahmadinejad and Hans-Rudolf Merz, his Swiss counterpart.
"This is not a break in relations, but an expression of Israel's discontent for the lax Swiss attitude towards Iran," an Israeli foreign ministry official told the AFP news agency.
Merz said criticisms of his meeting with Ahmadinejad on Sunday were "unjustified".
Israel also summoned Monika Schmutz-Kirgoz, the head of Switzerland's diplomatic mission to the country, for an "urgent discussion".
France has warned that European delegates would walk out of the meeting if Ahmadinejad makes "racist or anti-Semitic accusations".
"We will have to be very clear. We will not tolerate any slips," Bernard Kouchner, the French foreign minister, told France Info radio.
The Paris-based European Jewish Congress said in a statement that Ahmadinejad's presence meant the UN had "put the fox in charge of the hen house".
The speech by Ahmadinejad, who is a frequent critic of Israel and has cast doubt on the extent of the killing of Jews during the Second World War, coincides with Holocaust Remembrance Day in Israel, which begins at sundown on Monday.
Canada and Israel have said for months that they will shun the meeting, which the UN organised to help heal the wounds left by its last racism summit in Durban, South Africa, in 2001.
The US and Israel walked out of the conference after Arab states sought to define Zionism as being racist.
Washington cited "objectionable" language in a text prepared for the Geneva meeting as its reason for staying at home.
The draft does not mention Israel by name, but it reaffirms the Durban statement and its reference to the plight of Palestinians.
Barack Obama, the US president, announcing his administration's decision not to attend the conference, said Washington wanted a "clean slate" before tackling race and discrimination issues at the UN.
"If you're incorporating a previous conference that we weren't involved with, that raised a whole set of objectionable provisions, then we couldn't participate," he said at a news conference after the Summit of the Americas in Trinidad and Tobago on Sunday.
Navi Pillay, the UN human rights chief who is hosting the Geneva conference, said she was "shocked and deeply disappointed" by Obama's decision.
Pillay said that some countries were focusing solely on one or two issues to the detriment of the fight against intolerance.
Several Muslim nations at the summit in Switzerland have called for moves to prevent perceived insults to Islam, which they say have proliferated since the attacks on the US on September 11, 2001.