Former Taliban minister and 30 Taliban fighters reported dead in Afghan air strike.
No explanation was provided for the withdrawal.
A French newspaper, Le Journal du Dimanche, reported a week ago that a pullout of special forces in the southeast was in the offing.
It suggested the worsening security situation in Afghanistan was a possible reason for the decision.
James Bays, Al Jazeera’s correspondent in Afghanistan, said: “France’s decision is embarrassing for Nato … which has been seeking to boost the number of troops it has in Afghanistan.
“They [the special forces troops] are the only French troops in a combat role in Afghanistan.”
Alliot-Marie told France-Info radio station: “There is a general re-organisation of our troops.
“Among the planned changes is a withdrawal of special forces from Jalalabad, in the coming weeks.”
The French defence ministry said the troops would withdraw “at the start of 2007”.
Deployed in southeastern Afghanistan, the elite French troops have been involved in Nato’s fight against al Qaeda and the Taliban.
France has 1,700 troops, in addition to the special forces, in Afghanistan as part of the Nato-led international security assistance force.