Ahmadinejad had resisted the pressure for a week before the final blow appeared to be the public reading on state television of the order issued by Khamenei to dismiss Mashaie.
"The appointment of ... Esfandiar Rahim Mashaie as vice president is against your interest and the interests of the government and will cause division among your supporters," Khamenei said in a letter to the president.
"It is necessary to announce the cancellation of this appointment."
Mashaie's appointment sparked opposition from many in Iran, including some of the incumbent president's own supporters, more than a month after Ahmadinejad's win in disputed presidential elections.
The issue was also the topic of the main Friday prayer sermon. Ayatollah Ahmad Khatami, a senior religious leader, called on Ahmadinejad to follow the orders of Iran's supreme leader and dismiss Mashaie.
"We support the president ... but we never said that he has no flaws," Khatami told worshippers in his sermon in Tehran, the capital.
Khatami is a member of the Assembly of Experts, the body that dismisses and appoints Iran's supreme leader.
Meanwhile, a group of students held a demonstration in the streets of Tehran on Friday to protest against Mashaie's appointment, state media reported.
"Obeying the leader's order is the demand of the nation," they chanted.