The man, identified as Abd al-Fattah Salih, sprayed automatic fire during midday prayers at a mosque in the province of Dhammar, 60km south of the capital, Sanaa, a police official said.
Salih fled the scene, retreating to his house and refusing to turn himself in to authorities, the official said on customary condition of anonymity. Police stormed the house and Salih was killed in an exchange of gunfire, he said.
Investigations were under way, the official said. But he said Salih was not a wanted dissident and that a domestic dispute may have prompted the shooting.
Yemen, a conservative country at the southern tip of the Arabian peninsula, is Usama bin Ladin's ancestral homeland and has long been a hotbed of Islamist activity. It also is awash with weapons.
Authorities estimate there are some 60 million firearms - or three for each citizen - in the largely tribal country.
Last month, two people were killed and two were injured when a man detonated a hand grenade in a mosque in the southern province of Ibb.