Al-Barghuthi, who is running a distant second in opinion polls, was forced by three Israeli plainclothes occupation officers into a police car outside the Lion's Gate entrance to the walled Old City, witnesses said on Friday.
He was attempting to pray at the al-Aqsa mosque.
Al-Barghuthi flashed a 'V' sign for victory with his fingers before he entered the car. It was the second time al-Barghuthi had been detained by Jerusalem police during the campaign.
Israel has promised to allow free and fair Palestinian elections and agreed to limited campaigning in the disputed city. But it insists that only Palestinians with proper permits be allowed into the disputed city of Jerusalem. Al-Barghuthi lives in the West Bank.
Violating an agreement
Jerusalem police spokesman Shmuel Ben-Ruby said al-Barghuthi had violated an agreement not to enter the mosque compound. He said al-Barghuthi was detained for questioning.
The status of Jerusalem is one of the most contentious issues in the Israeli-Palestinian conflict. Israel captured east Jerusalem in 1967 and annexed it. The Palestinians want the eastern section of the city for the capital of a future state.
Jerusalem is one of the most
contentious issues in the conflict
Israel has agreed to allow east Jerusalem Palestinians to vote on Sunday. More than 5000 Palestinians will be permitted to vote in east Jerusalem, but tens of thousands more will have to travel to outlying suburbs to vote.
Al-Barghuthi is expected to receive only a small percentage of the vote. The leading contender, Mahmud Abbas, had cancelled tentative plans to visit the Old City in a campaign stop, fearing a large Israeli security presence could embarrass him among voters.
Al-Barghuthi was detained by Israeli police in the Old City last month, but released hours later. Israeli police had no immediate comment on Friday's arrest.