Egyptian Sayyed Imam Sherif, apparently an influential member of al-Qaida, was arrested on Wednesday in the Lawder area, one official said.
The announcement of Sherif's arrest followed one earlier of another suspected top al-Qaida leader Abd al-Rauf Nassib in the same operation.
"Sayyed Imam Sherif is also wanted by Egyptian police," the official said, adding that "he was the number one in Egypt's Islamic Jihad, and succeeded by Ayman al-Zawahiri," currently al-Qaida's number two.
The two men were captured at the same time, he said, after security forces backed by helicopters and armoured vehicles were deployed on Wednesday in the area where they surrounded "about a dozen" suspected Islamists.
The security forces were alerted after the Islamists, from Yemen, Algeria, Egypt and Saudi Arabia, had gathered in the Lawder region, 250km south of Sanaa and 150km northeast of Aden.
"Sayyed Imam Sherif is also wanted by Egyptian police"
The fugitives were facing a 48-hour ultimatum that expires on Friday "to surrender unconditionally or face a military assault", according to the main Yemeni opposition party, al-Islah.
"Sayyed Imam Sherif, alias 'Doctor Fadhel', gave up the Islamic Jihad leadership to Ayman al-Zawahiri in 1991, on the heels of dissidence within the movement, to devote himself to publishing religious studies", said another official.
"Sherif and Nassib were among 12 Islamists arrested on Wednesday," a security official meanwhile said, adding that the two men were hiding with "dozens of other suspects" in the mountainous Jebel Thira region, which is difficult to access and lies above the town of Lawdar.
The local authorities have refused to identify the 10 other suspects arrested or give details about the organisations they belong to.
Islamist sources named Nassib as one of the most senior al-Qaida officials in Yemen, and the sole survivor of a missile attack launched from a US drone aircraft that killed six suspected al-Qaida members in Maarib province of eastern Yemen on 3 November 2002.
Yemen's Islamic Jihad is suspected of links with Usama bin Ladin's al-Qaida network.