Iran's state television identified the dead man as the first secretary in Iran's embassy, Khalil Naimi.
"Khalil Naimi, first secretary of Iran's embassy to Iraq, who was in his diplomatic car, was attacked by unknown people and died immediately," the station's Baghdad reporter said.
The attack took place on Haifa street near the Iranian embassy on Thursday, the reporter added.
A Reuters correspondent on the scene saw a car with at least two bullet holes in it. A body was slumped in the vehicle, which had smashed into a lamp-post after the shooting.
"We have been told that he was driving his car to go to the embassy and three men drove up and shot him," an Iranian official said in Baghdad.
An Iranian delegation has been in Iraq to try and help mediate between the US-led authorities and Shia cleric Muqtada al-Sadr.
The US occupation administration has vowed to kill or capture al-Sadr and destroy his militia, but both sides have said they are eager to avoid bloodshed.
Under the rule of Saddam Hussein's Sunni Muslim-dominated secular government, some among Iraq's majority Shia community sought exile and developed close ties with Shia-ruled Iran.
Meanwhile, Iraq's leading Shia Muslim cleric Ayat Allah Ali al-Sistani has warned the US against entering the holy city of Najaf in pursuit of al-Sadr, Shia religious and political sources said on Thursday.
"The Ayat Allah has told the Shia Iraqi Governing Council members to inform the Americans that Najaf is a red line," a senior source told Reuters.
"The Najaf religious establishment is adamant against any US military action in the city," the source said.
A 2500-strong US force is near Najaf after soldiers were sent south from bases north of Baghdad.