Kharrazi said that Iran had not only not provided refuge to Al-Qaeda members escaping from Afghanistan, but it was actively fighting the organisation.
He accused the United States of allowing an armed Iranian opposition group to "stay and function in Iraq" even though it was "a registered terrorist organisation".
The foreign minister was referring to the Mujahedin Khalq which is on the US' list of terrorist organisations.
The Mujahedin Khalq was formerly supported by the ousted Iraqi government to conduct military operations against Iran.
"I think it is the United States that is harbouring terrorists, because right now every corner of Iraq is controlled by Americans," said Kharrazi.
Kharrazi made his comments during his visit to South Africa for a meeting of a South-African-Iranian bilateral commission.
His remarks came in response to US President George W Bush's threats earlier on Monday that Iran and Syria should mend their ways or suffer the consequences.
Bush accused the two states of supporting "terrorists" who undermine Middle East peace efforts.
But Kharrazi said: "If President Bush means by terrorists the al-Qaeda organisation, then not only are we not harbouring them, but we have arrested them and sent (them) back to their countries of origin and even put some of them in prison."
Bush said that Syria and Iran were continuing to harbour and assist terrorists.
"This behaviour is completely unacceptable and states that support terror will be held accountable," he said.
The US president was speaking at a news conference at his Texas ranch with Italian Prime Minister Silvio Berlusconi.
Berlusconi, who supported Bush's war on Iraq despite the wishes of his people, is in America to discuss the situation in Iraq.
Axis of evil
Bush has branded Iran - along with North Korea and the Iraq of ex-President Saddam Hussein - as a member of an "axis of evil" that threatens to spread terrorism.
Washington charges Iran is not doing enough against members of al-Qaeda network.
US officials have also accused Syria of backing Palestinian and Lebanese groups it classifies as terrorist, such as Hamas and Hizb Allah.
It is concerned their conflict with Israel could endanger the Middle East peace process.
Bush urged Middle Eastern governments including Syria and Iran to support the peace efforts of Israeli Prime Minister, Ariel Sharon, and Palestinian Prime Minister, Mahmoud Abbas.
"Both Prime Minister Abbas and Prime Minister Sharon are showing leadership and courage. Now it's time for governments across the Middle East to support the efforts of these two men by fighting terror in all its forms," Bush said.
Bush’s comments come a day after Iran announced that it has developed a ballistic missile that is capable of hitting Israel.
Both Syria and Iran accuse the US of being the real danger to world peace.