Security officials said five men died on the spot and three died of their wounds later.
The attack took place in Lahaj province south of the capital Sanaa.
The officials said the motive behind the attack was not immediately clear.
The sources said the gunman, who was arrested, could have had a grudge against moderate Muslims.
However, no one has yet claimed responsibilty for the shooting.
The Yemeni government, which has been under pressure from Washington to crack down on Muslim groups, routinely blames "Islamists" for violent attacks.
They have been blamed for assaults against Westerners in recent years, but clashes between tribes and groups are also frequent in a country where unlicensed weapons are plenty.
The shooting comes a day after the families of about 200 dissidents detained in Yemen on suspicion of terrorism links, appealed to President Ali Abdullah Saleh for their release.
"We ask you to release our children and Muslim brothers detained by the state security services," they wrote in a message to the president.
The families protested that "some of them have been kept in detention for more than two or three years ... without any specific accusation", while others were "suffering from illnesses and psychological problems".
In mid-July, the Yemeni interior minister told parliament about 200 suspects accused of links to attacks in Yemen would be brought to trial.
He also said that 95 others suspected of belonging to the al-Qaeda had been released after renouncing its ideology.