A member of the Japanese contingent press centre in Kuwait said on Friday: "Today is the beginning of the withdrawal. But it is not complete."
The pullout of about 550 troops from a military base in southern Iraq, where they have been engaged in reconstruction and humanitarian work, will end the Japanese military's riskiest and most ambitious overseas mission since World War Two.
Earlier on Friday, Reuters witnesses saw transport trucks leaving and helicopters flying out of the desert base in Samawa in the province of Muthanna.
Kyodo news agency said a C-130 transport aircraft carrying about 30 soldiers arrived at an airport in Kuwait on Friday.
Nuri al-Maliki, the Iraqi prime minister, said Iraqi forces would take over Muthanna security in July, the first Iraqi province outside peaceful northern Kurdistan to be handed over to Iraqis since the US-led invasion.
The British oversee a multinational contingent in Muthanna that includes Japanese and Australian troops.
Britain says the handover of security to Iraqi forces does not mean the withdrawal of its troops from the region.
They will stay on as a rapid reaction force to back up Iraq's fledgling security forces.
Japan's troop dispatch, a symbol of Tokyo's willingness to put "boots on the ground" for its close ally, the United States, has won praise from Washington.
But it was opposed by many at home including critics who said the dispatch violated Japan's pacifist constitution.