The withdrawal of 2500 soldiers on Thursday comes after the Free Aceh Movement (GAM) disbanded its military wing this week.
The Finnish-mediated pact ended one of Asia's longest running separatist wars.
It was signed in August after talks between the two sides accelerated following last December's Indian Ocean tsunami, which left 170,000 people dead or missing in Aceh.
Smiling soldiers carrying weapons and backpacks boarded several ships set to depart from the port city of Lhokseumawe.
Lieutenant Anugerah, from the East Java city of Surabaya, who has been in Aceh for eight months, said: "I'm very happy. I have missed my wife terribly."
The final withdrawal of Indonesian security forces - a contingent of police reinforcements - will take place on Saturday.
Aceh military commander Supiadin AS said in a speech sending off the troops that the government had now pulled 24,125 soldiers from Aceh under the 15 August peace agreement.
Finnish-officiated talks ended in
a peace agreement in August
He did not say how many were left, but once the last police contingent goes, government forces in Aceh should be no more than 14,700 soldiers and 9100 police.
For its part, GAM has handed in 840 weapons.
The success of the weapons handover and the almost simultaneous withdrawal of tens of thousands of troops and police has surprised even optimists.
The 30-year war has killed some 15,000 people, mostly civilians.
Security experts say the next challenges are finding jobs for demobilised GAM fighters and satisfying rebel demands for political participation in Aceh.
The tsunami disaster created pressure for the two sides to end their conflict and open way for a massive international recovery and rebuilding programme.