"I'm sure this official must be referring to Mingora city rather than the Swat valley because Swat will take more time," he told Al Jazeera by phone from Rawalpindi.
"The main valley is likely to be completed, let's say, in two weeks time."
He said that the valley itself "is quite spread out and it will warrant the clearance of all those [Taliban] positions in the countryside [and] up in the mountain ranges".
Analysts also questioned the defence secretary's statement.
"What he is trying to say is that, as far as the battle is concerned, it's going to be over very soon," Talat Masood, a defence analyst and retired general from the Pakistan army, told Al Jazeera.
"But in such like insurgencies, it's not just winning the battle.
"As you regain territory, you have to make sure that the administrative structures are set up."
The debate over how long it will take Pakistan to re-take the Swat valley, come a day after the military said it had regained full control of Mingora, the valley's main city.
Imran Khan, Al Jazeera's correspondent reporting from Islamabad, the capital, said the defence secretary's assessment of victory in two to three days was "an optimistic statement".
"It depends in part on what your indicator is - whether you think it is routing the Taliban that means victory or whether its repatriating 3.4 million refugees and giving them back their homes and their land," he said.
About 2.4 million people have fled in the wake of the military's campaign against the Taliban, joining people displaced by earlier fighting.
There are no figures of civilian casualties, but some of the displaced have told of innocent relatives being killed.
Around 300,000 people lived in Mingora until the Taliban occupied the town in early May when the army first launched an offensive in Swat.
Also on Sunday, there were reports that at least 15 fighters and between two and three soldiers were killed in clashes in South Waziristan, in Pakistan's North West Frontier Province.
Fighting there has intensified in recent days amid the offensive in the nearby Swat valley and there is an expectation that the military will turn its focus on the area, which borders Afghanistan, once its Swat operation is over.