An Iraqi army captain said the militia was withdrawing.
"They started pulling out early this morning and they're still going," he said.
Shops began to reopen in Diwaniyah on Tuesday and water and electricity supplies were turned back on, as a tense calm returned to the town 180km south of Baghdad.
Iraqi army soldiers fought a 12-hour battle with al-Mahdi Army fighters in Diwaniyah on Monday, after the Shia militia began bringing reinforcements into town and took control of several neighbourhoods.
The al-Mahdi Army is a loosely-organised militia group loyal to the Shia cleric Muqtada al-Sadr.
Under the deal brokered by Iraqi political leaders, the army will pull back reinforcements which came from outside the city, while the al-Mahdi Army will evacuate a district it took control of during the fighting.
Iraqi officials have also agreed to re-examine the case of an al-Mahdi Army commander who was arrested at the weekend, the Iraqi army captain said.
Al-Sadr supporters outside Diwaniyah have blamed Tuesday's fighting on "infiltrators" working without their leader's approval.
Mohammed al-Askari, spokesman for the defence ministry, said 20 soldiers and 40 militiamen had been killed in the fighting.
A security official, speaking on condition of anonymity, said many of the soldiers were shot at close range, execution-style after being captured by militia fighters.
The head of Diwaniyah's health department, Hamid Taathi, said that his hospitals had received the bodies of eight civilians and treated 61 bystanders for wounds received in the fighting.
Polish soldiers from the US-led coalition in Iraq set up blocking positions around Diwaniyah to prevent militia reinforcements joining the battle, but were not directly involved in the fighting, a US military official said.