"The idea is that reconciliation is absolutely essential in Iraq - I don't think anyone would argue with that," Annan said on Saturday, as he made his first visit to Baghdad since the US-led invasion in 2003.
Annan's day-long visit took place under tight security and coincided with a car bombing in the capital that killed four people and injured 40.
Annan also paid tribute to his colleagues who were killed in the car bombing of the UN headquarters in Baghdad in August 2003.
Speaking at a news conference with Iraq Prime Minister Ibrahim al-Jaafari, the UN chief spoke out in favour of national reconciliation and urged all factions to be allowed to join in the political reform process.
"The political transition must be a process that is inclusive and transparent and takes into account the concerns of all groups," he later added.
Annan, who met a number of government leaders and Sunni Arab representatives, also spoke out against terrorism, terming it "absolutely unacceptable".
The attack on the UN building
in August 2003 killed 22 people
"No ideology, no cause can justify the killings," he said.
Annan's visit came one day after US Secretary of State Condoleezza Rice made similar appeals for reconciliation on a visit to Iraq.
On a visit to the UN headquarters in Baghdad, Annan addressed UN staff and spoke kindly of his envoy Sergio Vieira de Mello, who was among 22 people killed in a truck bombing at the former UN headquarters in August 2003.
"I have been wanting to come for quite some time," he said.
"As I walked into our building I stopped by a monument erected to our dear friends who died here in 2003... They carried no guns, they came to help and their lives were cut short," he told reporters inside the heavily guarded UN compound in central Baghdad.
The United Nations has been operating at greatly reduced levels in the country since international staff were withdrawn in October 2003 after two bombings at its Baghdad offices but its operations are slowly beginning to expand.