Yemeni Foreign Minister Abu Bakr al-Qirbi said on Wednesday the two countries had agreed at summit talks in Riyadh to step up joint control of the border.
The Saudi government suspects weapons have flowed across the border to al-Qaida members in the country.
"It has been agreed the construction will be suspended and the two sides will be responsible for coordinating activities for patrolling borders to prevent smuggling and illegal migration of personnel," al-Qirbi said.
Yemen had complained the Saudi fence violated a border treaty signed by the two countries in 2000 which established a demilitarised zone on either side of the frontier.
Al-Qirbi said Saudi Arabia had built nearly 75km of barrier in separate sections along the 1300km border with Yemen.
The barrier, in parts a concrete-filled pipeline and in others a sand barrier, fell within 100 metres of the border in some areas, he said.
Under their 2000 border agreement, which established the precise line of the border after decades of dispute, Saudi Arabia and Yemen agreed to a demilitarised zone of between 5km and 20km on both sides of their border.
Al-Qirbi said the Riyadh talks, which began late on Tuesday and were continuing on Wednesday, had been "very cordial".
"Both sides had maintained their position on fighting terror," he said.
They also agreed to "establish a mechanism for the control of border smuggling between Saudi Arabia and Yemen".