Up to 75,000 Iraqi and US soldiers using tanks and armoured vehicles were to be deployed across the capital starting on Wednesday in "Operation Forward Together".
Nuri al-Maliki also announced plans for an extended curfew and a weapons ban.
Not long after the clampdown started, a car bomb killed one person and wounded five, a police source said.
In Kazimiyah, a northern area of the city, about 2,000 supporters of Muqtada al-Sadr, the Shia cleric, protested against the surprise visit of the US president on Tuesday.
Hazem al-Araji, an al-Sadr aide, condemned the "ill-fated visit" by George Bush, "the leader of the occupation", and protesters chanted "Iraq is for the Iraqis".
At one point, they tried to burn an American flag, but it failed to ignite.
Major General Mahdi al-Gharrawi, the commander of public order forces under the interior ministry, said the security operation was to be the biggest of its kind in Baghdad since the US handed over sovereignty to Iraq in June 2004.
"Baghdad is divided according to geographical area, and we know the al-Qaeda leaders in each area," he said.
A bomb went off on Wednesday
despite the clampdown
Al-Gharrawi also said he believed that fighters opposed to the US presence in Iraq were likely to step up their attacks.
"We are expecting clashes will erupt in the predominantly Sunni areas," he said.
Civilians have complained of random violence and detentions by Iraqi forces, especially the police, which are widely believed to have been infiltrated by so-called sectarian death squads.
Al-Gharrawi said there were plans for a single uniform to distinguish legitimate forces in the coming days.
"There will be a special uniform with special badges to be put on the vehicles as a sign that it belongs to our forces," he said.
Al-Maliki's plan includes banning personal weapons and implementing a 9pm to 6am curfew, which hitherto had begun at 11pm. The new curfew was expected to begin on Friday.
There would be more house raids, more checkpoints and even air strikes in the city.
An Iraqi soldier mans his gun in
the Kadhimiya area of Baghdad
Al-Malaki said: "The raids during this plan will be very tough ... because there will be no mercy towards those who show no mercy to our people."
The Iraqi army launched a similar crackdown dubbed Operation Lightning in May last year, deploying more than 40,000 Iraqi police and soldiers, backed by American troops and air support.
However, violence continued to increase and many Sunnis were alienated by the heavy-handed tactics concentrating on their neighbourhoods.