The demonstration, called by the Iraqi Islamic Party, followed Friday prayers and was staged outside the Green Zone, a heavily protected area which is home to the Iraqi US-backed government, the US army and the US embassy.
The zone was the site of the Iraqi republican palace, built in the 1960s.
Protesters, who included many women, carried banners urging the authorities to respect human rights and acted out scenes depicting Iraqi soldiers torturing a Muslim cleric.
They also called on the Iraqi government and the US army in Iraq to release Iraqi detainees. Reports suggest that thousands of Iraqis are being detained for indefinite periods and without trial.
"It is shameful to see our innocent fellow citizens being arrested, tortured and even killed by some in the security services which claim to be acting in the name of the law," Aala Makki, an Islamic Party official, told the crowd.
And he called on the government to set up a commission of inquiry to look into alleged violations and to "purge security forces" of "those who try to make trouble between Iraqis".
"Use of force doesn't solve problems but makes them worse, and state terrorism is not an answer," Shaikh Ayad al-Azzi told the faithful during Friday prayers, before the demonstration.
Iraqi President Jalal Talabani said this week that government officials had opened investigations into several cases of alleged police and military abuse, but gave no details on the cases. And US officials have acknowledged that reports of police brutality are a matter of concern.
Hundreds of former Iraqi Special Forces demonstrated in Baghdad on Thursday, calling for their reemployment.
The 600 soldiers of al-Shahwani unit were dismissed from the service some months ago, and were replaced by new recruits. Protestors of the unit, which was named after the current Iraqi spy chief Muhammad al-Shahwani, said they demonstrated to get their voices heard.
"Al-Shahwani told us that we are like his brothers and he would not let us down, we are sure that our dismissal was done behind his back" a protestor said.
"They told us we are American agents," another frustrated protestor said.