Annan was speaking on Friday after a UN Security Council meeting on the implementation of Resolution 1559, which calls for Syrian troops and intelligence agents to leave Lebanon immediately, and demands the disarming of militia.
Hizb Allah leader Shaikh Hasan Nasr Allah on Monday repeated the stance that the UN resolution did not apply to his group because it is not a militia but a resistance movement.
Annan said that since Syrian forces had been asked to withdraw, the disarmament of the militia would have to be carried out by the Lebanese government.
"Hopefully the groups concerned will agree to disarm themselves and cooperate with the Lebanese authorities because it is not going to be an easy task if it has to be
done by force or imposition," he said.
Nasr Allah said Hizb Allah was
a resistance movement
"I would hope governments that have influence in the region and can help in the process would do so."
His envoy, Terje Roed-Larson earlier on Friday gave a report to the Security Council on the implementation of Resolution 1559.
"It was the Lebanese civil war that led to the deployment of foreign forces on Lebanese terrority. Now, 30 years after the eruption of the civil war, a full and complete Syrian withdrawal from Lebanon would represent a significant and important step towards drawing a final line under the saddest chapter in Lebanese history," he said.
Syria sent troops to its smaller neighbour in 1976 to help quell what was then a year-old civil war, but the troops remained after the war ended in 1990.
But Damascus came under relentless international pressure to withdraw which intensified after the 14 February assassination of former Lebanese prime minister Rafiq al-Hariri.
"I would hope governments that have influence in the region and can help in the process would do so"
Annan said he believed the team he sent earlier this week to verify the withdrawal of Syrian troops would be able to do the job.
The United Nations would monitor the upcoming elections in Lebanon, the first round of which are set for 29 May, the secretary-general said.
Annan has said any delay in the elections would threaten the security, stability and prosperity of the country.
On Monday, officials from UN Security Council members will discuss their reaction to Annan's report on the implementation of Resolution 1559.