Many carried banners, decrying the killing of women, workers, academics and scientists.
Different Shia groups have been wrestling for control of Basra, Iraq's second-largest city and the urban centre of an oil-rich region.
Residents are becoming increasingly alarmed about security, saying that killings, kidnappings and other crimes have increased significantly since British forces turned over responsibility for the city at the end of last year.
In February, two journalists working for CBS were kidnapped in Basra. One was released but the other, a Briton, is still being held.
Dozens of women were slain in Basra last year because of how they dressed - their mutilated bodies found with notes warning against "violating Islamic teachings".
Saturday's protesters, who were mostly men, came from several Shia political movements, including the biggest Shia party, the Supreme Islamic Iraqi Council and its militia wing, known as the Badr Brigade.
Khalaf said at a news conference later that "today's demonstration was a natural right of the citizens and the political parties to express their opinions".
He defended the performance of the police, saying they had freed 10 people who were kidnapped in the past 10 days and "detained 64 people accused of carrying out sabotage and terrorist operations all over Basra".
The protests came as Iraq witnessed more violence.
Separate roadside bombings killed six people in Wajihiya, about 25km east of Baquba.
In the first attack, a bomb destroyed a car - killing a mother and her two children and wounding two others, including the woman's husband.
The second attack hit a bus, killing three men and wounding two others, said a security official, who spoke on condition of anonymity.
Also on Saturday, a mass grave containing about 100 bodies was discovered near Khalis in Diyala province, about 80km north of Baghdad.
Colonel Sabah al-Ambaqi of the Iraqi police said the grave was discovered in an orchard near al-Bu Tumaa, a Sunni village outside Khalis.
Khalis is a Shia town surrounded by Sunni communities and has been the scene of repeated sectarian attacks..