Afghan and British troops exchanged fire with opposition fighters two kilometres from the centre of Musa Qala on Sunday.
The forces captured two Taliban commanders, a statement from Afghanistan's defence ministry said.
Residents flee
Azimi said between 200 and 300 residents had fled Musa Qala.
He said the town had become a base for "foreign terrorists" and that "hundreds of terrorists had massed there".
A Taliban commander inside Musa Qala said there were up to 2,000 fighters in the area ready to defend themselves.
Mullah Hafizullah, apparently referring to civilians, said: "We also have almost the same number of friends."
Owen Fay, Al Jazeera's correspondent in Kabul, said on Sunday that the Taliban have no intention of leaving Musa Qala.
"From what we understand, the warnings from military spokesmen and the Afghan government are going unheeded," he said.
"A Taliban spokesman had said in the past that Musa Qala wasn't overwhelmingly significant for their operations and that their fighters could simply be deployed elsewhere.
"But commanders on the ground in Musa Qala say they have no intention now of surrendering whatsoever and that they will fight to the death."
Meanwhile, the US-led coalition, which works alongside the Afghan army and Nato's International Security Assistance Force (Isaf), said it had dropped a precision-guided bomb on a fighter compound in Musa Qala district on Friday, killing several Taliban fighters including a commander.
Locals fleeing
"Hundreds of locals are fleeing the town while the poorest and most vulnerable people have had to stay behind," Nazanine Moshiri, Al Jazeera's correspondent in Afghanistan, said.
Moshiri said the Taliban had said that they had captured five Afghan soldiers during the fighting and later killed them.
Gul Ahmad Khan, a 55-year-old resident of the town, said he and his family were packing up a few items and planned to leave for a safer area.
He said the troops were about three kilometres from the town centre.
"They have come to the villages surrounding the town but have not come into the town so far," Khan said.
Another resident, who gave his name only as Sharrafuddin, also said troops had not yet entered the town but were in outlying villages.
The Taliban took control of Musa Qala in February. The town and the region around it have seen heavy fighting this year.
A deal which saw British troops hand control of the area back to tribal elders lasted only a few months before the Taliban returned.