"The day of vengeance is near and your strong towers in the Green Zone will not protect you," said the statement posted on a web site often used by Islamist militants and signed by the new leader Abu Hamza al-Muhajir.
The Green Zone in Baghdad is a heavily fortified compound in Baghdad which houses the headquarters of the Iraqi government, foreign embassies and a US military base.
The appearance of the web statement coincided with a surprise visit to Baghdad on Tuesday by George Bush, the US president, to try to bolster the Iraqi government days after al-Zarqawi was killed in a US air strike.
The authenticity of the statement could not be verified. It would be al-Muhajir's first public statement since being named the new leader of al-Qaeda in Iraq.
"Coming battles will reveal the falseness of your power and the cowardliness of your soldiers," al-Muhajir said. "Do not rejoice that you killed (al-Zarqawi), he has left behind lions that ... trained under him.
"You will see what we have in store for you because of your betrayal and apostasy. Our swords are poised above your necks," he told Iraqis who cooperate with US-led forces.
On Monday, the group named the little-known al-Muhajir to be al-Zarqawi's successor.
A US counterterrorism official who spoke on condition of anonymity said the latest threat came as no surprise.
"It's in large part propaganda but no one's being dismissive of it. One of the things they're trying to demonstrate is that they will continue the jihad and that they're still a force to be reckoned with," he said.
"They're trying to assure the al-Qaeda rank-and-file that they're going to continue fighting and remain viable."
Al-Muhajir also addressed part of his statement to al-Qaeda leader Osama bin Laden saying: "We await your signal and follow your orders, and we give you the good tidings that morale is high among your soldiers".