The victims were gathered together, Sunnis were then separated according to the names on their identity cards and killed after about an hour, said Maitham Abd al-Razzaq, an Iraqi police lieutenant.
An Iraqi interior ministry official, speaking on condition of anonymity for security reasons, said Shia militiamen wearing masks and black uniforms also roamed through the area, grabbing people from the street.
An imam at a local mosque told Aljazeera that the number killed could be as high as 50.
Police were searching for more victims reportedly dumped in the streets.
The attack was apparently in retaliation for the car bombing the night before of a local Shia mosque that left two dead.
US and Iraqi forces sealed off the area and began searching for the attackers, Abd al-Razzaq said.
Salam Zikam Ali al-Zubaie, the Iraqi deputy prime minister, called the attack "a real and ugly massacre".
He said responsibility for the killings lay with Iraqi security forces that are widely believed to have been infiltrated by Shia militias.
"There are officers who instead of being in charge should be questioned and referred to judicial authorities," al-Zubaie told Aljazeera.
Shia cleric Muqtada al-Sadr called on the government and Iraqis to exercise restraint.
In a statement, he also called on all parties to meet to stop what he called a "Western plot" to incite a sectarian war and urged the Iraqi parliament to hold an emergency session.
Ibrahim al-Shammari, spokesman of the Islamic Army in Iraq said he believed that the danger posed by the armed militias outweighed that of the danger posed by foreign troops.
He demanded that Shia tribal chiefs take a stand on sectarian violence in Iraq.
Clashes also broke out in northwestern Baghdad between US forces and members of the al-Mahdi army, the militia loyal to al-Sadr, leaving three militia members dead, police said.
In other violence on Sunday, an Iraqi army intelligence officer in the Shia city of Karbala was shot and killed.
Major Qahtan Adnan Abd al-Razzaq, with the Iraqi army's al-Hussein Brigade, was shot dead after his car was intercepted in the centre of Karbala, 80km south of Baghdad, said Salim al-Abadi, a health official.
Two policemen were also killed in gun attacks in eastern Baghdad and the northern city of Kirkuk, police said.