Junichiro Koizumi was due to hold a news conference at 1pm (0400 GMT) on Tuesday, having conveyed the decision on the withdrawal, which media reports said would begin as early as this month, to his junior coalition partner earlier in the day.
Nuri al-Maliki, the Iraqi Prime Minister, said on Monday his forces would take over security from July in the southern province of Muthanna, where the British oversee a multinational contingent that includes Japanese troops.
Japan's troop dispatch - a symbol of Tokyo's willingness to put "boots on the ground" for its close ally, the United States - won praise from Washington but was opposed by many at home.
The decision to withdraw comes before Koizumi's visit to Washington for talks with President George Bush in late June.
Kazuo Shii, leader of the opposition Communist Party of Japan, has said that Koizumi will consider expanding its air force operations in Iraq.
"Koizumi told me that he would like to consider expanding the activities of the Air Self-Defence Force," Shii told reporters after the meeting.
No Japanese soldiers have been killed or wounded in Iraq, but Koizumi faced a difficult political situation in 2004 when three Japanese civilians were taken hostage by an armed group.
The three, as well as two others taken hostage later, were released unharmed.
In all, six Japanese citizens, including two diplomats, have been killed by fighters opposed to the presence of US-led troops in Iraq.
About 550 Japanese troops are
based in Samawa in the south
Taro Aso, the Japanese foreign minister, said the troops had won high marks for their military discipline.
"I think such views have made a very big contribution to improving the brand image of Japan as a country," he told a news conference before the official announcement.
Japan has said its withdrawal from the southern city of Samawa will have to be co-ordinated with the British and Australian governments, whose troops have been providing security for the roughly 550 Japanese non-combat troops there.
Japan's military activities overseas are limited by its pacifist constitution, although the government has been stretching those restrictions in recent years.
Australia troops re-assigned
In Canberra, Brendan Nelson, the defence minister, said on Tuesday that Australia would re-assign its 460 troops protecting Japanese forces to help the Iraqi military to secure the border with Syria.
"It has the potential to be more dangerous for our soldiers. We don't underestimate the risk," he told reporters.
Howard faces calls to bring
Australian troops home
The announcement has sparked calls to bring the troops home.
They will provide back-up and training for Iraqi forces who are set to take control of the southern province of al-Muthanna, and help secure the dangerous Syrian border, Nelson said.
The move is politically sensitive for John Howard, the prime minister's government, which backed the US-led offensive in Iraq in the face of widespread public opposition.
Koizumi had stressed the need to back the United States in Iraq to ensure that Washington comes to Japan's aid in the event of an attack by its neighbour, North Korea.
North Korea fears
The expected announcement coincides with assessments by some officials that North Korea may be poised to launch a missile that some experts say could reach as far as Alaska.
After the withdrawal of ground troops, Japan's air force is expected to expand its transport activities in Iraq from a base in neighbouring Kuwait.
Speaking at a news conference on Monday, Koizumi gave no timetable for a withdrawal but said Japan would continue to provide reconstruction and humanitarian aid to Iraq.
Japan decided to send ground troops to Iraq in December 2003, and the first major contingent arrived in February 2004.
Since then, they have been engaged in reconstruction work such as repairing buildings and providing medical training.