Rejecting any comparison with Israel's controversial West Bank barrier, Saudi border guard chief Talal Anqawi told the pan-Arab newspaper Asharq al-Awsat on Monday that he plans to prevent vehicles crossing into the kingdom illegally.
But preventing lucrative arms smuggling in mountainous tribal areas is likely to prove a huge task.
Tonnes of weapons, ammunition and explosives have been seized over the years, but is likely to represent only a small percentage of arms that actually make their way north.
The two countries, who only recently ended a long-running border dispute, have stepped up security cooperation since suicide bombings in Riyadh killed more than 50 people last year.
But Yemeni newspapers have reported that the government has complained about the route of the barrier.
"What is being constructed inside our borders with Yemen is a sort of screen...which aims to prevent infiltration and smuggling. It does not resemble a wall in any way"
Saudi chief border guard
Sanaa says the barrier is being built in a 20km zone designated as an open area under a 2000 border agreement.
The chief secretary to the Border Commission, Muhsin Ramadan, told Aljazeera.net on Monday that Sanaa had sent a delegation to Jidda to protest the Saudi decision.
However Anqawi said the barrier was being built on Saudi soil and denied any comparison of the screen to the barrier being built by Israel.
"What is being constructed inside our borders with Yemen is a sort of screen...which aims to prevent infiltration and smuggling. It does not resemble a wall in any way."
Riyadh is also close to awarding a contract worth up to seven billion euros to defence electronics company Thales.
The French firm hopes to have the contract to supply a border surveillance system within weeks.
Pictures of the Saudi-Yemeni barrier reveal a section of raised, concrete-filled pipeline similar to that on the kingdom's northern border with Kuwait.
Diplomats say Saudi Arabia is urgently stepping up border controls after the surge of domestic violence and instability last year.