The move on Wednesday comes a day after China cancelled a visit to Beijing by Japan's foreign minister amid a newly released public opinion poll that indicated that the Japanese are evenly split in support of Koizumi's pilgrimage despite neighbouring countries' outcry.
Koizumi sparked protests from Beijing and Seoul on Monday when he prayed at Tokyo's Yasukuni shrine, which critics say glorifies Japan's militaristic past and the invasion of Asia.
The visit complicates relations already strained over territorial disputes and Japan's approval of textbooks that opponents say whitewash the country's second world war era atrocities.
South Korean Foreign Minister Ban Ki-moon said on Wednesday it would be inappropriate for him to travel to Japan, where he was reportedly planning to visit in the next two weeks, although no date had been set.
"It's true that there have been consultations through diplomatic channels, but the schedule hasn't been fixed," Ban told his weekly news briefing.
"In the current situation, however, the atmosphere is inappropriate to push for a visit to Japan."
South Korea's presidential office also said it would be "difficult" for President Roh Moo-hyun to hold summit talks with Koizumi in December as scheduled "unless there is a significant change in the situation".
About 42% of Japanese back the leader's most recent visit, while 41% said he should not have gone, according to a poll released on Wednesday by the Asahi newspaper.
The findings suggest deep divisions at home over an issue that has driven a wedge between Tokyo and the governments in China and South Korea.
South Korea's Ban Ki-moon (R):
Inappropriate to visit Tokyo now
Japanese Foreign Minister Nobutaka Machimura tried to contain the diplomatic fallout on Tuesday, but his efforts were overshadowed as nearly 200 other Japanese lawmakers and aides followed Koizumi's example by paying their own visits to the shrine.
The Korean Peninsula was ruled as a Japanese colony from 1910 to 1945, and resentment against Tokyo runs deep.
Japan invaded China in 1937 and is blamed for the massacre of as many as 300,000 civilians in the eastern city of Nanjin.