But Obama, who is on a visit to the Middle East and reached Cairo from Saudi Arabia, cautioned against expecting immediate change.
"No single speech can eradicate years of mistrust, nor can I answer in the time that I have all the complex questions that brought us to this point," he said.
"But I am convinced that in order to move forward, we must say openly the things we hold in our hearts, and that too often are said only behind closed doors."
Before moving forward, the world must address the effects of extremism, he said.
"The first issue that we have to confront is violent extremism in all of its forms," Obama said, repeating his declaration in Ankara in April that "America is not - and never will be - at war with Islam."
"We will, however, relentlessly confront violent extremists who pose a grave threat to our security.
"Because we reject the same thing that people of all faiths reject: the killing of innocent men, women, and children. And it is my first duty as president to protect the American people."
Obama also acknowledged that one major issue of tension remains the Israeli-Palestinian conflict.
He said US-Israeli relations are strong and "unbreakable" and are based on the recognition that the aspiration for a Jewish homeland is rooted in a tragic history that cannot be denied.
But he said it is also undeniable that the Palestinian people have "suffered" in pursuit of their own homeland.
"The only resolution is for the aspirations of both sides to be met through two states, where Israelis and Palestinians each live in peace and security.
"That is in Israel's interest, Palestine's interest, America's interest, and the world's interest."
He said Palestinians must renounce violence, while Israelis must halt settlement construction and ensure that Palestinians can live, work and develop their society.
Marwan Bishara, Al Jazeera's senior political analyst, said it helped undo "the harm done by the Bush administration".
"If Bush had to demonise many Muslims in order to launch the wars he did in the Islamic world, then Obama humanises the Islamic world in order to engage," Bishara said.
However, Bishara said, Obama was not forthcoming enough on a number of issues on US involvement in the Middle East, such as Israel's nuclear arsenal.
And Obama's call for Arab states to normalise relations with Israel and that Arabs must accept a Jewish state in Palestine remained part of the Israeli narrative on the Middle East conflict, Bishara said.
|The US president spoke before a crowd of invited guests at Cairo University [Reuters]
A spokesman for Mahmoud Abbas, the Palestinian president, described Obama's speech as a "good start".
"His call for stopping settlement and for the establishment of a Palestinian state, and his reference to the suffering of Palestinians ... is a clear message to Israel that a just peace is built on the foundations of a Palestinian state with Jerusalem as its capital," Nabil Abu Rdainah said.
"President Obama's speech is a good start and an important step towards a new American policy," he said.
Ahmad Yousuf, a senior Hamas official, told Al Jazeera that Obama's speech reminded him of Martin Luther King's "I have a dream speech".
About Obama stressing on the legitimacy of Israel, he said the Palestinians must have a state of their own before being asked to recognise another.
However, Fawzi Barhoum, a Hamas spokesman, added: "He did not respect the democratic choice of the Palestinian people who voted for Hamas."
Obama also addressed the war in Afghanistan in his speech.
He said the United States does not want to keep its troops in Afghanistan, and is not seeking military bases there.
But he said the US is in the South Asian nation out of necessity, following the attacks of September 11, 2001.
"Let us be clear: al-Qaeda killed nearly 3,000 people on that day. The victims were innocent men, women and children from America and many other nations who had done nothing to harm anybody.
"They [al-Qaeda] have affiliates in many countries and are trying to expand their reach. These are not opinions to be debated; these are facts to be dealt with."
The US leader also broached the subject of Iran's controversial nuclear programme, reaffirming America's "commitment" to seeking a nuclear weapons-free world.
"No single nation should pick and choose which nations hold nuclear weapons... And any nation, including Iran, should have the right to access peaceful nuclear power."
Obama's trip to the Middle East is part of efforts to build a coalition of Muslim governments to support his efforts to revive stalled Israeli-Palestinian peace talks and help the US curb Iran's nuclear programme.