Abe's support rates have already been slashed by public outrage over government mishandling of pension records.
Kyuma is the second minister to resign since Abe became prime minister last September.
Abe has selected Yuriko Koike, Japan's national security adviser, as Kyuma's replacement, according to Japanese media reports.
Offence caused
An estimated 74,000 died in Nagasaki,
and 140,000 in Hiroshima [AP]
Kyuma, who represents Nagasaki in the lower house of Japan's parliament, made his offending statement in a speech in Chiba on Saturday.
"I understand that the bombings ended the war, and I think that it couldn't be helped," he said.
He said that although the atomic bombings of Nagasaki and Hiroshima were painful, they saved Japan from continuing its fight at a time when the Soviet Union was threatening to take territory in northern Japan.
Kyuma's statement contradicted the Japanese stance, fiercely guarded by survivors and their supporters, that the use of nuclear weapons is never justified.
A ban on possession of such weapons is a pillar of Japan's postwar pacifist regime.
US decision questioned
In January, Kyuma raised eyebrows in Washington by calling the US decision to invade Iraq a "mistake" because it was based on the false premise that Saddam Hussein possessed weapons of mass destruction.
Japan and the US are close military allies, and Japan hosts about 50,000 American troops under a security treaty.
On August 6, 1945, the US dropped a bomb nicknamed 'Little Boy' on Hiroshima, killing at least 140,000 people in the world's first atomic bomb attack.
Three days later it dropped another atomic bomb, 'Fat Man', on Nagasaki, where about 74,000 are estimated to have been killed.
Japan surrendered on August 15, 1945.