The interior ministers of both countries signed the accord on Friday at a meeting of the Yemeni-Saudi Coordination Council in Al-Mukalla, Yemen.
"The signing of this agreement is a demonstration of the political good will between the two countries' leaderships," said Saudi Crown Prince Sultan Bin Abdel Aziz.
The Saudi authorities fear that Islamist radicals blamed for a series of attacks in recent years have used the porous 1,800km border to smuggle arms and explosives.
In April 2005, Saudi Arabia began constructing a concrete wall along the border to prevent the smuggling of drugs including qat - a stimulant popular in Yemen.
Yemeni officials objected to the building of the wall, saying it was being erected in a demilitarised area that should remain open, according to the 2000 border agreement.