Al-Rabai was killed after anti-terrorism troops carried out a dawn raid on a house in the capital, Sanaa, based on a tip, security officials said.
The forces killed another man, Muhammad al-Dailami, also suspected of being an al-Qaeda member. One soldier was wounded in the raid.
A security official said: "Rabai was killed in a shootout with security forces in a suburb of Sanaa, where he was hiding in a house."
Al-Rabai, a Yemeni, was sentenced to death in 2004 for his role in the attack on the French tanker and planning to kill the American ambassador to Yemen and bomb five Western embassies.
The Limburg attack left one crew member dead and 12 injured.
Al-Rabai and 22 others escaped from prison in February.
The escapees included Jamal al-Badawi, who was convicted of plotting, preparing and helping to carry out the October 12, 2000, attack on the destroyer USS Cole in which 17 American sailors were killed in the Yemeni port of Aden.
Of the 23, nine have surrendered and security forces have killed four. Ten remain at large.
Two bombers rammed an explosive-laden boat into the Limburg, killing a Bulgarian crew member and spilling 90,000 barrels of oil into the Gulf of Aden.