The UN chief was responding to a question about the disarmament of Hizb Allah, which showed its strength on Tuesday at a huge pro-Syrian rally in Beirut attended by hundreds of thousands of people who chanted anti-US slogans.
Two huge banners read in English: "Thank you Syria" and "No to foreign interference".
Annan said the world needed to accept that in every society different groups may hold different views.
"Of course, we need to be careful of the forces at work in Lebanese society as we move forward," he said.
"But even the Hizb Allah - if I read the message on the placards they are using - they are talking about non-interference by outsiders ... which is not entirely at odds with the Security Council resolution, that there should be withdrawal of Syrian troops," Annan said.
"But that having been said, we need to recognise that they are a force in society that one will have to factor in as we implement the resolution," he said.
The rally by Hizb Allah vastly outnumbered anti-Syrian rallies of the past weeks.
The Syrian-backed Lebanese resistance group, which is funded by Iran, is the best armed and best organised faction in Lebanon and enjoys strong support among Lebanon's Shia Muslim community.
"We need to be careful of the forces at work in Lebanese society as we move forward"
United Nations secretary-general
Many of the signs at the rally in Riad al-Sulh square denounced UN Security Council Resolution 1559, which calls for Syrian troops and intelligence agents to leave Lebanon immediately and demands the disarming of militias, referring to Hizb Allah.
Syrian soldiers entered Lebanon in 1976 to try to quell a civil war that began the previous year. They remained through 14 years of fighting that ended in 1990, and about 14,000 are still there, though they have started pulling back to the border.
Annan declined to discuss the timetable for withdrawal, saying he sent Terje Roed-Larsen, his top envoy on the Syria-Lebanon issue, to talk to top officials in Beirut and Damascus this week about the pullout and was awaiting his return.
"After his discussion, then I will know better how we are going to proceed," Annan said. "I am going to give a report to the council in April. I hope I will be able to report progress."
Roed-Larsen will hold talks with
officials in Beirut and Damascus
The secretary-general was asked whether the United Nations was considering sending a force to Lebanon after the Syrian withdrawal to ensure security. The world body has a peacekeeping force in southern Lebanon, but it has no mandate to operate elsewhere in the country. "I've read in certain newspapers that the UN may have to send in a force to monitor the withdrawal of the Syrian troops, but I have no such mandate as of today," Annan said, adding that he was not involved in any discussions about a UN or international force for Lebanon.