Mohamed Vall, Al Jazeera's correspondent in Yemen, said al-Raimi was a prominent member of al-Qaeda who escaped from a Yemeni jail in 2006.
Ayed al-Shabwani was the leader of al-Qaeda in Maarib, east of Sanaa, our correspondent said.
Security officials told news agencies that al-Shabwani had provided a hideout and training facilities at his farm in the mountainous eastern province.
The latest reported deaths come after Abdullah Mehdar, the purported head of an al-Qaeda cell in the eastern province of Shabwa, in an attack on the house where he was hiding.
Several other figures, who the security forces say are senior al-Qaeda members, have also been killed or arrested in recent weeks.
Yemen has intensified operations against the so-called Al-Qaeda in the Arabian Peninsula group claimed it was behind a failed December 25 attempt to blow up a Detroit-bound US airliner.
Gregory Johnsen, a Yemen scholar at Princeton University in the US state of New Jersey, said al-Remi's death, if confirmed, would be "a very, very significant blow against Al-Qaeda in the Arabian Peninsula".
"He is the most dangerous individual in the group based on his history, his statements and his role," he told the Reuters news agency.
"He fought in Afghanistan and was central to rebuilding the organisation and planning and conducting attacks."
Yemen's government declared war on the al-Qaeda movement in the country on Thursday and warned its citizens against aiding the network.
"The campaign to track down al-Qaeda terrorists will continue [and] the government will use all means at its disposal to eradicate terrorism from Yemeni territory," an official said.
Earlier this week, troops were sent to three provinces to join operations against al-Qaeda fighters.
The government in Sanaa is fighting the so-called Houthi rebel group in the north of the country and a secessionist movement in the south, as well as al-Qaeda.