An official from the interior ministry said Muhammad Hamdi al-Ahdal was arrested after police surrounded a house in the capital Sanaa on Tuesday.
Yemeni authorities have accused al-Ahdal - also known as Abu Issam al-Makki - of taking part in several attacks on Western interests in the country.
He has been pursued for nearly two years and is believed to be one of the top two commanders of al-Qaida's Yemeni branch.
Several attacks on Western targets have taken place in Yemen in recent years, notably the 2000 bombing of the US warship Cole and the 2002 attack on the French oil tanker Limburg.
Since then, the Yemeni government of President Ali Abd Allah Salih has worked closely with the United States to crack down on armed groups such as al-Qaida.
More than 20 people were arrested last month on suspicion of planning attacks on Western and local targets.