He denied any wrongdoing, but the accounting scandal was widely perceived to be a major reason for Abe's Liberal Democratic party's devastating defeat in Sunday's elections for parliament's upper house.
"There is no doubt about the cause of the ruling party's election loss. I feel very sorry, and I have decided to step down," Akagi said on national television news on Wednesday.
"There is no doubt about the cause of the ruling party's election loss. I feel very sorry..."
Norihiko Akagi, agriculture minister
Since late last year, three other ministers have left their jobs.
In May, Akagi's predecessor, Toshikatsu Matsuoka, committed suicide amid allegations of misuse of public funds.
In July, Fumio Kyuma, Japan's foreign minister, quit following an uproar over his remarks that the Hiroshima and Nagasaki atomic bombings in 1945 may have been justified.
Late last December, Genichiro Sata, the administrative reform minister, stepped down over charges of misusing political funds.
Abe's cabinet has been hit by money scandals, ministers' gaffes and a fiasco over the loss of pension records for 50 million citizens.
Japanese anger was reflected in a telephone poll conducted by the English-language Asahi Shimbun which showed public support for Abe's cabinet hitting an all-time low since he took office.
In the survey released on Wednesday, figures for those unhappy with the cabinet climbed from 56 per cent to 60 per cent.
Abe, however, has refused to step down despite one of the worst showings for his party at the polls.
The Asahi poll also found that public sentiment was divided over whether Abe should stay on.
A poll by Yomiuri Shimbun, Japan's largest newspaper, had similar results.
Hidenao Nakagawa, the Liberal Democratic secretary-general, said: "We should make transparency in matters of money and politics a basic condition in forming the next cabinet to prevent a recurrence."