The UN secretary-general told a news conference in Saudi Arabia on Monday: "The two sides have accepted the effort of the secretary-general to help solve this problem.
"I will appoint a person to work secretly with the two sides ... I will not announce his name today or tomorrow."
The release of two Israeli soldiers, whose capture by Hezbollah in a cross-border raid on July 12 led to a 34-day war, is at the centre of a UN resolution for a permanent ceasefire on the Lebanon-Israel border.
Hezbollah wants to exchange the soldiers for Lebanese prisoners held in Israeli jails. Both Hezbollah and Israel have been involved in prisoner exchanges in the past.
Annan said: "the only [thing] I insisted on is that if I am going to use my good offices, then my mediator should be the only mediator.
"If [others] get involved, we will pull out because you would get wires crossed."
Word of the mediation mission came as Annan met Saudi Arabia's King Abdullah in the Red Sea port of Jedda, the latest stop in his 11-day tour of the Middle, with the intention of getting all sides to implement and support the resolution.
Israel's official position remains that it will not enter into any negotiations to free the soldiers, Mark Regev, the Israeli foreign ministry spokesman, said.
An Israeli government official said: "A mediator is not needed.
"The UN resolution determines that soldiers will be released unconditionally. The UN secretary-general will assist and not mediate."
However, a top Israeli political source said last week that Israel would be willing to discuss freeing Lebanese prisoners for the soldiers if negotiations were conducted through the Lebanese government and not with Hezbollah.