Yemen's government has been battling al-Qaeda in the country at the same time as dealing with a Shia uprising in the north and rising secessionist sentiment in the south.
The rally took place in the southern Abyan province in an area that was bombed in an air raid last week.
The government said at the time that the raid had foiled a planned series of suicide bombings by attacking targets that included an al-Qaeda training centre.
But dozens of civilians, including children, are thought to have been killed in the bombing.
Abbas al-Assal, a local human rights activist, said at the time that 64 people were killed, including 23 children and 17 women.
Abdul-Ghani al-Iryani, a political analyst based in Yemen, said that the air raids meant civilian casualties were unavoidable, but that the government could take steps to lessen public anger.
"Unfortunately collateral damage cannot be avoided in operations like this [as] al-Qaeda live with their families in their bases and training camps, so there's no way of avoiding it," he told Al Jazeera.
Al-Iryani said the government should announce the names of the al-Qaeda operatives killed in the raid.
"There's nothing that the government can do in regards to dealing with al-Qaeda per se, but addressing people's grievances in other issues could increase the credibility of the state," he said.
The New York Times newspaper said on Saturday that the US gave military hardware, intelligence and other support to Yemeni forces to carry out the attack.
Supporters of Yemen's so-called Southern Movement, which calls for independence and accuses the government in Sanaa of sidelining the south, participated in the rally which marked the bombing, calling for an investigation into the attack.
An explosion disrupted the rally and killed three people. The cause of the blast was unclear, though it has been blamed on a cache of unexploded munitions.
Fighters in Yemen and Saudi Arabia earlier this year announced a loose alliance under the name of "al-Qaeda in the Arab Peninsula", taking Yemen as their base.