"Only five to 10 per cent of the job is remaining and hopefully within two to three days, the pockets of resistance will be cleared," he was quoted as saying.
Ali, who made his comments at a defence conference in Singapore, is Pakistan's third highest ranking defence official.
But Pakistan's military moved quickly play down his comments.
"I'm sure this official must be referring to Mingora city rather than the Swat valley because Swat will take more time," Abbas told Al Jazeera.
Analysts had also questioned the defence secretary's statement.
"What he is trying to say is that, as far as the battle is concerned, it's going to be over very soon," Talat Masood, a defence analyst and retired general from the Pakistan army, told Al Jazeera.
"But in such like insurgencies, it's not just winning the battle. As you regain territory, you have to make sure that the administrative structures are set up."
The debate over how long it will take Pakistan to re-take the Swat valley, comes a day after the military said it had regained full control of Mingora, the valley's main city.
Imran Khan, Al Jazeera's correspondent reporting from Islamabad, the capital, said the defence secretary's assessment of victory in two to three days was "an optimistic statement".
"It depends in part on what your indicator is - whether you think it is routing the Taliban that means victory or whether its repatriating 3.4 million refugees and giving them back their homes and their land," he said.
About 2.4 million people have fled in the wake of the military's campaign against the Taliban, joining people displaced by earlier fighting.
There are no figures of civilian casualties, but some of the displaced have told of innocent relatives being killed.
Around 300,000 people lived in Mingora until the Taliban occupied the town in early May when the army first launched an offensive in Swat.
Also on Sunday, there were reports that at least 15 fighters and between two and three soldiers were killed in clashes in South Waziristan, in Pakistan's North West Frontier Province.
Fighting there has intensified in recent days amid the offensive in the nearby Swat valley and there is an expectation that the military will turn its focus on the area, which borders Afghanistan, once its Swat operation is over.