At least 10 people were injured and another 15 detained, Al Jazeera's correspondent in Jerusalem said.
Israeli police put the number of arrests at 12.
Early on Sunday, Israeli police deployed extra troops after calls for demonstrations around the holy site.
The Palestinian calls came amid rumours that rightwing Jewish activists were planning to gather at al-Aqsa compound.
The rumours circulated after a fringe Israeli group, the Organisation for the Defence of Human Rights on the Temple Mount, called on Jews to gather at the mosque compound as well as the adjacent Western Wall.
Palestinian officials said the Israeli police closed off the compound to visitors, leaving hundreds of worshippers inside.
Jivara al-Budairi, Al Jazeera's correspondent in Jerusalem, said: "Clashes between Palestinians and the Israeli police spread to the Old City neighbourhoods of Bab Hutta, Bab al-Majlis and Aqabat al-Tkiye.
She said the injured could not be moved out of the compound because of the Israeli police siege.
She said the violence in the Old City erupted after the Israeli police fired tear gas and stun grenades at Palestinian students and youths in the area.
The youths retaliated by throwing stones at the soldiers.
The neighbourhood's merchants have, meanwhile, announced a comprehensive strike and closed all shops, al-Budairi said.
Micky Rosenfeld, the Israeli police spokesman, said the raid and arrests took place after Palestinian youths threw stones and a petrol bomb at a police patrol near the mosque.
Shmuel Ben-Ruby, the Jerusalem police spokesman, admitted that security forces used stun grenades to disperse the demonstrators.
He accused the protesters of pouring oil on the ground to make the police forces slip, and of hurling a firebomb.
Ben-Ruby said police did not enter the mosque itself.
But Kamal Khatib, a spokesman for the Israeli Arab Islamic Movement, which has been at the forefront of recent al-Aqsa demonstrations, blamed Israeli police for the clashes.
"The police always excuse their attacks by saying that the worshippers threw stones," he told the AFP news agency.
"It is clear they just want to justify their crimes."
Khatib further accused the police of stopping buses filled with Muslim worshippers in northern Israel in a bid to prevent them from reaching Jerusalem.
Describing the situation to Al Jazeera, Sheikh Muhammad Hussein, the mufti of Jerusalem, said: "Israeli forces entered the compound from the Maghareba and Silsila gates.
"The forces cordoned off the compound, preventing all Muslims from entering the mosque.
"The situation is extremely serious, and I expect it to escalate.
"The Israelis have beaten the mosque's guards and staff, as well as worshippers, including women.
"I was personally prevented from entering the mosque. They are preventing us by force from reaching the mosque, where Sheikh Abdul Azeem Salhab, head of the endowments council, several endowments department staff and a number of worshippers are at present under siege.
"We are besieged in al-Haram al-Sharif yard, while they are trapped inside al-Aqsa mosque.
“A large number of Israeli policemen and security officers are deployed inside al-Haram al-Sharif. They have detained most of the Palestinian worshippers who were inside the compound."
Jacky Rowland, Al Jazeera's correspondent in East Jerusalem, said: "Palestinians living in the occupied part of the city put up with a lot of indignity and harassment on a daily basis - demolitions, evictions, checkpoints.
"But when it comes to anything that threatens the integrity of al-Aqsa mosque, that is where people's patience snaps and that is why we have seen such an angry response all over East Jerusalem [from people] who see this as a very heavy display of police might."
Tensions exploded into violence on September 27, when Palestinians hurled rocks at a group of visitors whom they suspected of being rightwing Jewish extremists.
Israel captured the compound from Jordan in the 1967 Middle East War and it has since served as a symbol of the two sides' competing claims to Jerusalem.
Day-to-day administration of the site remains in Muslim hands.
In September 2000, the second Palestinian uprising, or intifada, erupted after Ariel Sharon, a rightwing politician who went on to become Israel's prime minister, visited the site.