An Israeli police spokesman said at least 16 people were arrested, but that calm had largely returned to the area several hours after the clashes broke out.
Israeli police action
Jivara al-Budairi, Al Jazeera's correspondent in Jerusalem, said the violence in the Old City erupted after the Israeli police fired tear gas and stun grenades at Palestinian students and youths in the area.
The youths retaliated by throwing stones at the police.
Israeli police had deployed extra troops to the site early on Sunday after Palestinians called for demonstrations in response to rumours that rightwing Jewish activists were planning to gather at the compound.
The rumours circulated after a fringe Israeli group, the Organisation for the Defence of Human Rights on the Temple Mount, called on Jews to gather at the mosque compound as well as the adjacent Western Wall.
A spokesman for Mahmoud Abbas, the Palestinian president, condemned "the storming of Haram al-Sharif by Israeli forces" and called on Israel to "halt all provocative acts".
"Jerusalem is a red line that cannot be crossed," Nabil Abu Rudeina told the AFP news agency, calling on the international community to intervene to "put pressure on the Israeli government".
Palestinian officials said the Israeli police had closed off the compound to visitors, leaving hundreds of worshippers inside.
Shmuel Ben-Ruby, the Jerusalem police spokesman, said security forces used stun grenades to disperse the demonstrators.
He accused the protesters of pouring oil on the ground to make the police forces slip, and of hurling a firebomb.
Ben-Ruby said police did not enter the mosque itself.
But Kamal Khatib, a spokesman for the Israeli Arab Islamic Movement, which has been at the forefront of recent al-Aqsa demonstrations, blamed Israeli police for the clashes.
"The police always excuse their attacks by saying that the worshippers threw stones," he told AFP.
"It is clear they just want to justify their crimes."
Jacky Rowland, Al Jazeera's correspondent in East Jerusalem, said: "Palestinians living in the occupied part of the city put up with a lot of indignity and harassment on a daily basis - demolitions, evictions, checkpoints.
"But when it comes to anything that threatens the integrity of al-Aqsa mosque, that is where people's patience snaps and that is why we have seen such an angry response all over East Jerusalem [from people] who see this as a very heavy display of police might."
Tensions had exploded into violence earlier on September 27, when Palestinians hurled rocks at a group of visitors who they suspected of being rightwing Jewish extremists.
Israel captured the compound from Jordan in the 1967 Middle East War and it has since served as a symbol of the two sides' competing claims to Jerusalem.
Day-to-day administration of the site remains in Muslim hands.
In September 2000, the second Palestinian uprising, or Intifada, erupted after Ariel Sharon, a rightwing politician who went on to become Israel's prime minister, visited the site.