Seoul's rejection of the proposed talks was confirmed by Nobutaka Machimura, the Japanese chief cabinet secretary, who said it was unlikely that the Japanese and South Korean foreign ministers would meet for talks.
"I think it is important for responsible people in charge of diplomacy to have candid discussions especially at a time like this," he said.
Japan had offered to hold a bilateral meeting at the Asean (Association of Southeast Asian Nations) regional forum in Singapore.
Seoul and Tokyo have long been at odds about who has sovereignty over the rocky outcrops, known as Dokdo to South Koreans and Takeshima in Japan.
|Korea has inhabited the islands since 1952 [EPA]
Earlier this week Japan said it plans to teach students about its claim to the islands, triggering an angry response from South Korea who recalled its ambassador to Tokyo and bolstered security around the disputed region.
On Thursday South Korean demonstrators clashed with police as they tried to push their way into the Japanese embassy.
Angry protesters battered, disembowelled and beheaded live pheasants, while dozens of war veterans in military fatigues shouted "Dokdo is our territory!" as they ate the birds' internal organs and dripped blood on Japanese flags and on pictures of present and former Japanese leaders.
There were also banners that read "Stop violating our territorial sovereignty!" and "Japan must stop distortion of history".
In April, Japan and South Korea held their first full-fledged bilateral summit in three years.
Ties were suspended after repeated visits by earlier Japanese leaders to a Tokyo shrine honouring the country's war dead including convicted war criminals.
Kwon Chul-hyun, South Korea's ambassador in Tokyo, has called on Japan to "take corrective measures" to normalise what he called "the collapsed diplomatic relations" between the two countries.
"It's not long ago that we shook hands with each other to make a fresh start," he said.
"The most desirable measure is to delete the description from the teaching guidelines."
South Korea's Yonhap news agency quoted Kwon as saying that the issue may also affect cooperation with Japan in six nation talks on disarming North Korea.
"If public opinion worsens at home or political circles strongly oppose such cooperation, we have no other choice but to take it into consideration," he was quoted as saying.
Last weekend, the two Koreas, China, Japan, the United States and Russia agreed to work towards the disablement of North Korea's nuclear plants by October.