France and the United States were the major players behind a policy statement which was read at a formal meeting in New York on Tuesday.
The security council, which also called on Secretary-General Kofi Annan to report on any progress twice a year, intended to bolster a 2 September resolution demanding troops be withdrawn.
Both Syria and Lebanon ignored last month's request, saying it was an interference in their internal affairs.
Tuesday's statement required the consent of all 15 council members.
Algeria, the only Arab member of the council, hesitated until the last minute.
Annan's last report said Syria had neither complied with the 2 September resolution nor given any timetable for doing so.
Lahud's presidential term has
been extended by three years
It also said Lebanon had not disarmed "militias", a reference to Hizb Allah and Palestinian groups, as called for by the resolution.
Shortly after the announcement, Syrian Foreign Minister Faruq al-Shara rejected the security council's call.
"Our position remains as it was -[Resolution] 1559 is illegal intervention in the Syrian-Lebanese bilateral relationship."
Syria's government has maintained political control over Lebanon since it intervened in 1976, at the request of the country's Christian Maronite community, to quell a civil war.
The issue came to the fore in September, when the UN Security Council tried unsuccessfully to keep Lebanon's parliament from amending the constitution and extending Syrian-backed President Emile Lahud's term for three years.