Khamenei criticised US behaviour towards Iran since its 1979 Islamic revolution, saying that the US was "hated in the world" and should stop interfering in other countries' internal affairs.
He asked how Obama could congratulate Iranians on the new year while the US continues to accuse the country of supporting terrorism and seeking nuclear weapons.
Saying that a change of US words was not enough, Khamenei said: "We will watch and we will judge ... You change, our behaviour will change."
The US is at odds with Iran over its nuclear programme, which Washington says is aimed at building atomic weapons, while Tehran insists it is for the peaceful generation of electricity.
It is also concerned over Iran's missile-development efforts.
The US cut off diplomatic ties with Iran during the 1979-1981 hostage crisis, in which a group of Iranian students held 52 US diplomats hostage at the American embassy for 444 days.
Khamenei said that tests at the Bushehr nuclear plant in the south of the country had convinced other nations that its nuclear plans would not be abandoned.
"This is the results of the progress of our scientists ... which persuaded the whole world that the path of Iran's nuclear progress could not be blocked," he said.
Obama released a video message on Nowruz, the traditional Persian New Year, offering to resolve past conflicts.
"My administration is now committed to diplomacy that addresses the full range of issues before us, and to pursuing constructive ties," he said in the message.
Obama's administration earlier expressed an openness to face-to-face diplomatic contacts with Tehran.
It marks a major shift from the policy pursued by George Bush, Obama's predecessor, towards Iran, which he once branded as part of an "axis of evil".
In his video appeal, Obama said the US wanted Iran to take its "rightful place in the community of nations".
"This process will not be advanced by threats," he said. "We seek instead engagement that is honest and grounded in mutual respect."
|Obama said US wants Iran to take its rightful place in the community of nations [Reuters]
But Obama also insisted that Tehran do its part to achieve reconciliation.
"You have that right - but it comes with real responsibilities, and that place cannot be reached through terror or arms, but rather through peaceful actions that demonstrate the true greatness of the Iranian people and civilisation," he said.
"The measure of that greatness is not the capacity to destroy, it is your demonstrated ability to build and create."
Commenting on Khamenei's rejection of Obama's offer, Mohammad Marandi, a Tehran university professor, told Al Jazeera: "It was expected. Iran has many grievances. A change in tone is not really enough.
"He [Khamenei] said if the US changes its policy ... especially towards Israel and the Palestinian people, the Iranians can move forward.
"I think if the Americans do make a significant move the Iranians will be willing to look towards better relations".