The farmers' group, in his constituency in northern Japan, has not yet returned the money to state coffers.
Shinzo Abe, the Japanese prime minister, named a new cabinet on Monday after his initial cabinet was plagued by scandals and gaffes.
The ruling coalition suffered a disastrous defeat in a July 29 election that gave the opposition a majority in parliament's upper house.
Abe's first appointee to the farm portfolio committed suicide over a separate scandal. The second was fired over reports of discrepancies in his political funding records.
Two other ministers in Abe's first cabinet, which was formed a year ago, were also forced to resign for gaffes or scandals.
Abe's public support ratings rebounded to around 40 per cent according to some media surveys after he named the new cabinet packed with political veterans.
But doubts about his leadership capability remain.
Endo apologised to the public on Saturday, but said he did not intend to resign.
Mizuho Fukushima, head of the tiny opposition Social Democratic Party, said Endo should step down.
"He should first fulfil his responsibility to explain, and then he should resign," he said.
"If he does not, we will pursue this with a censure motion."
Opposition parties have the votes to pass a censure motion in the upper house. The motion would not be binding, but would be a major embarrassment and would put enough pressure on Endo to resign.
Naoto Kan, a senior executive of the main opposition Democratic Party, also criticised Endo's actions and threatened to call him to testify in parliament.
"The explanation given by the farm minister is not at all persuasive to the public nor to the Democratic Party," Kan said.
Separately, pressure came from Endo's ruling camp peers.
Kaoru Yosano, chief cabinet secretary, said: "He should explain in a more understandable manner if an explanation is sought."
The head of the junior partner in the ruling coalition sounded a note of angry frustration over the fresh scandal.
"Frankly, it is shameful and regrettable that such problems keep emerging one after another," Akihiro Ota, leader of the Buddhist-backed New Komeito, said.