Annan made the comments in Istanbul on Monday, as he met scholars, politicians and religious leaders to discuss ways to improve relations between 'East' and 'West'.
"We should start by reaffirming and demonstrating that the problem is not the Quran or the Torah or the Bible," Annan said in response to a report by the multinational group of scholars that sets out proposals to overcome the problem.
"Indeed, I have often said that the problem is never the faith, it is the faithful and how they behave towards each other."
The 'Alliance of Civilisations' was jointly formed by Spain and Turkey in 2005 as part of a UN-backed initiative which was led by the Spanish government in response to the Madrid train bombings.
Middle East peace
Annan also said that any effort to stop growing violence between Islamic and Western societies must include an end to the Arab-Israeli conflict.
"We may wish to think of the Arab-Israeli conflict as just one regional conflict amongst many. It is not. No other conflict carries such a powerful symbolic and emotional charge among people far removed from the battlefield," he said.
Annan said he would work along with his successor, Ban Ki-moon, to help implement the recommendations of the report, which called for renewed efforts toward the goal of establishing "two fully sovereign and independent states co-existing side by side in peace and security."
"As long as the Palestinians live under occupation, exposed to daily frustration and humiliation, and as long as Israelis are blown up in buses and in dance halls, so long will passions everywhere be inflamed," Annan said.
The report was drafted over the past year by a group of 20 prominent men and women, which includes Mohammad Khatami, the former Iranian president, and Archbishop Desmond Tutu, the South African activist and Nobel Peace Prize winner. Both were in attendance at the ceremony held in Istanbul.
The UN initiative is co-sponsored by the prime ministers of Spain, a predominantly Catholic country, and Turkey, which is 99 per cent Muslim.