The 73-year-old was caught at a farmhouse outside his home village of Corleone in the Sicilian countryside, officials said.
Provenzano was Italy's most wanted man and was believed to have taken over as head of the mafia in 1993 after the arrest of Toto 'the beast' Riina.
Police said he was being transported to a jail in the Sicilian capital, Palermo.
Provenzano's disguises and ability to evade capture were legendary, and the last picture police had of him before his arrest was taken in 1959.
In April 2003, police arrested a Palermo man they were sure was Provenzano, but turned out to be a doorman who resembled a composite image of him.
Provenzano managed to communicate from his Sicilian hideout through a series of hand-delivered notes on tiny pieces of paper known as 'pizzini'.
In January police arrested 46 people in Sicily in an effort to disrupt Provenzano's support network.
However, as recently as last month, his former lawyer was quoted as telling an Italian newspaper that the elusive man was dead.
"I think he (Provenzano)'s dead, and has been dead for several years," Salvatore Traina was quoted as telling La Repubblica.
"They have looked for him everywhere, they have looked intensely for years but they can't find him. This must mean something."
He has been sentenced in absentia to life in prison in connection with the Mafia's most notorious crimes in recent years, including the 1992 killings of anti-Mafia magistrates Giovanni Falcone and Paolo Borsellino.
As a young man he was known as "Binni the tractor" because of the way he mowed down the enemies of the Corleone clan.