The separate states of North and South Yemen came together in a fragile political agreement in 1990.
However, many southerners in this impoverished country of 19 million people continue to talk about marginalisation.
Witnesses said speakers at the rally complained that thousands of former government officials and soldiers from the south lost their jobs after the 1994 fighting, when some southern leaders went into exile.
"There are some obscure ideas around about turning the clock back," Abd-Rabbu Mansour Hadi, Yemen's vice-president, told a government rally in Aden, capital of the former South Yemen.
"Yes, there is financial and administrative corruption and yes, there are negative points. But we have to deal with them within the framework of Yemeni unity."
Somali pirates have in recent months staged numerous raids on ships passing through the Gulf of Aden, a key shipping lane for oil and cargo, making millions of dollars and causing US and other governments to send patrol ships to the area.
Saudi Arabia, the world's biggest oil exporter, fears that instability could allow Yemen to become a staging post for reviving a 2003-2006 campaign by al-Qaeda fighters to topple the US-allied ruling Al Saud family.
Yemen has faced al-Qaeda-related violence, a Zaidi Shia uprising and lawlessness among tribes over the past few years.