"We will carry on our mandate that has been given to us by the Security Council," he said at a news conference at Stockholm airport shortly before he was due to board a flight to New York.
Annan said the Security Council would meet later on Wednesday to discuss its next moves, but that there could be no question of pulling out of Iraq.
"We will not be deterred or distracted by this senseless act of terrorism," he said. "Yesterday was a dark Tuesday for the UN, Iraq and international solidarity. On that day the United Nations lost some of its most outstanding public servants, including Sergio Vieira de Mello," said Annan, with tears in his eyes.
In Baghdad there was speculation that the UN will transfer all its foreign employees in the city to Jordan.
Some UN staff in Baghdad said an evacuation is planned, but spokesman Salim Lone said no decision had been made.
A UN security officer told Reuters: "All UN international employees who worked at the Canal Hotel are going to be sent to Amman today. Five planes are expected at the Baghdad airport."
Vieira de Mello: The world reacted
to his death with shock and anger
Brazilian Vieira de Mello was the most senior UN official killed in a Middle East attack since Folke Bernadotte, a Swedish UN mediator, was gunned down by Zionists in September 1948.
Vieira de Mello struggled for survival for a few hours in the wreckage of his office, which took the full force of the blast, before he died.
Investigators believe he was a deliberate target since the bomb went off directly under his office.
The death of the 55-year-old Brazilian diplomat triggered shock and anger throughout the world, with his home country declaring three days of mourning.
US President George Bush said the bomb blast would not derail US efforts to rebuild Iraq.
He said: "The terrorists want to return to the days of torture chambers and mass graves... All nations of the world face a challenge and a choice."
The president announced no new measures to improve security after the attack, but pledged to provide "all possible assistance" to the UN rescue and recovery effort.
"Iraq is on an irreversible course toward self-government and peace. And America and our friends in the United Nations will stand with the Iraqi people," he added.
But in the Arab world speculation is rife about who perpetrated the attack and why they targeted the UN.
Some argue the UN is only in the country to help Iraq get back on its feet, while others say it is legitimising the American occupation.
Hassan al-Bazzaz, a professor of politics at Baghdad University, told Aljazeera the Iraqi people would be negatively affected by the consequences of the attack.
He said the UN would continue carrying out its mission in Iraq, as De Mello had an honest plan to save Iraq from its disastrous predicament.
But Muhammad al-Duri, the former Iraqi ambassador to the UN, said the American occupiers were responsible for the attack and for Vieira de Mello's killing.
"Without the occupation, this man who served the whole world for 30 years, would not have been killed," he said.
Washington will try to take advantage of the UN official's death by minimising the role of the world body in Iraq, he added.
US military officials said the Baghdad blast occurred when a cement truck loaded with about 227kg of explosives detonated.
Occupation officials said they suspected it was a suicide attack because body parts of the person believed to be driving the truck were found 200 metres from the crater.
No group has yet claimed responsibility for the blast.