Israeli police cleared the Maghreb Gate after a disturbance on Monday, Israeli public radio reported.
The protest comes amid a dispute about Israel's control over who can access the mosque compound.
Israeli authorities had reopened the mosque site to Israeli and foreign tourists last week.
The Muslim authority in charge of the site, or Waqf, has said it was looking to reopen the compound to help the Old City's struggling tourism industry but said the Israeli government should not take a unilateral decision.
But it also said any decision to reopen the compound to foreigners should be accompanied by a lifting of restrictions on Muslims.
Muslims under the age of 40 are currently barred from entering the site.
The compound is the third holiest site in Islam but Jews claim that it is the site of the second temple and since 1967 have appropriated a section of the site for worship.
In July 1995 the Israeli High Court of Justice ruled that Jews could pray at Al-Aqsa sparking widespread protests among Muslims.
Some of the passengers on a Jerusalem bus blown up last week were returning from a visit to the Western Wall, also known as the Wailing Wall, the longest wall bordering the compound and holy to Jews.
In late July, Israeli police suspended visits to the site by non-Muslims which had resumed several weeks earlier for the first time since the beginning of the Palestinian Intifada in September 2000.
The Intifada, or uprising, erupted following a visit to the mosque compound by current prime minister Ariel Sharon when he was still opposition leader.