In unusually tough language, Annan - in a letter obtained by the media on Friday - also criticised the Palestinian president for failing to meet obligations under a US-backed peace "road map", including security reforms and putting an end to human bombings.
He also said Israel had to carry out its road map requirements to dismantle Jewish settler outposts and freeze construction in larger, established settlements.
"You are aware that the Palestinian side too has obligations it has not fulfilled," Annan told Arafat.
"The Palestinian Authority should immediately start taking effective measures to curb terrorism and violence."
"Decisive actions on your part would help the international community ensure that any withdrawal from Gaza is part of the implementation of the road map and not a substitute for it," Annan wrote.
Annan's letter, received on Thursday, was a response to Arafat's written complaint on 15 April about Israeli Prime Minister Ariel Sharon's unilateral "disengagement" plan and US President George Bush's endorsement of it.
Bush angered Palestinians and the Arab world by giving Sharon unprecedented support for holding on to some occupied West Bank land and barring the return of Palestinian refugees.
The Palestinian leader criticised
Israel's disengagement plan
Sharon's proposal, which faces a crucial referendum of his right-wing Likud party on Sunday, calls for evacuation of all settlements in Gaza and four of 120 enclaves in the West Bank.
Palestinians fear Sharon is trying to trade Gaza for effective annexation of large swathes of West Bank land they want for a state.
Ram Allah headquarters
Meanwhile, Aljazeera's correspondent in Ram Allah reported that Israeli special forces were deployed on the vicinity of Arafat's headquarters.
"The Palestinian Authority should immediately start
taking effective measures to curb terrorism and violence"
"The occupation forces were seen in the area and the districts neighbouring the headquarters in Ram Allah", he added.
Also on Friday, the Israeli forces demolished 11 houses near al-Tufah crossing in the Gaza southern city of Khan Yunis.
The Israeli occupation forces at dawn on Thursday arrested the leader of the Aqsa Martyrs Brigades in Ram Allah, Ali al-Barguti in an ambush in al-Amari refugee camp.
"The occupation forces stormed al-Faria refugee camp blocked all roads leading to it and occupied a number of houses," said our correspondent.
"The occupation soldiers opened fire and killed a Palestinian woman and wounded a number of Palestinians including Aljazeera cameraman Nabil Mazawi," he added.