He visited Mecca in December 2005 for an Organisation of the Islamic Conference (OIC) summit but outside of the hajj season.
It is not clear whether he performed the pilgrimage privately before becoming president.
Relations have been rocky between Shia majority Iran and Sunni majority Saudi Arabia, which also has a substantial Shia community in its oil-rich eastern province.
Relations reached an all-time low in July 1987 when 402 people were killed in clashes between Iranians and Saudi security forces during the Hajj.
However, Ahmadinejad's appearance is seen as a sign of warming relations between the two countries.
Ahead of this year's Hajj, Iran urged Saudi Arabia to crack down on religious extremism following reports of anti-Shia sermons and pamphlets in the kingdom.
Last month, Ahmadinejad publicly rebuked King Abdullah for Saudi Arabia's participation in a meeting in the US city of Annapolis aimed at finding peace between Israel and the Palestinians.
But he also became the first president of Iran to attend the end-of-year annual summit of Gulf Arab leaders that was held in Qatar earlier this month.