The men were put overnight on a government bus bound for South Africa, according to their lawyer Jonathan Samkange.
He said he had not had contact with the men, but had received confirmation they had left prison.
Marge Pain, wife of one of the group, Kenneth Pain, said she knew for sure the men were on their way. She said Zimbabwean officials had told relatives, who waited through the night at the border, that the men were coming under heavy police escort, according to the South African Press Association.
Journalists were not allowed near the South African border crossing, and there was confusion about the whereabouts of the men.
South African radio said there were conflicting signals from the authorities. It reported that only six of the men apparently had valid passports and that the rest were likely to be detained by immigration officials and transported to an undisclosed location.
The alleged ringleader Mann (L)
will serve a four-year sentence
The men, who all have South African nationality although some are of Angolan or Namibian origin, were arrested in March 2004 when their aging chartered plane landed in Harare on the way to oil-rich Equatorial Guinea. The Zimbabwean authorities charged them with plotting to overthrow the government.
During a lengthy trial last year, they denied being part of a coup plot and said they were bound for Congo to work as security guards at a diamond mine.
The Zimbabwean court convicted them of relatively minor immigration charges after prosecutors failed to prove more serious weapons and coup conspiracy charges.
The men completed a one-year prison sentence on Tuesday and were formally released on Thursday, but were forced to stay in the top security Chikurubi prison compound, because of bureaucratic bungling and apparent delaying tactics by the authorities.
The men's lawyers said Zimbabwean officials gave a series of excuses - ranging from security concerns during the journey to South Africa to the need to buy film for identity pictures - to delay their release.
Mark Thatcher pleaded guilty to
bankrolling the plot
Samkange said the South African authorities were also to blame. He said some of the men had not been given valid passports after their own documents had expired while in jail.
"Their [the South African embassy] argument was that they had not been approached to provide passports," said Samkange on Saturday.
The South African authorities had refused to intervene despite appeals from the lawyers and relatives.
South Africa is embarrassed about its reputation as a ready source of mercenaries, many of whom used to fight in
apartheid-era defence forces.
Some of the men held in Zimbabwe were Angolans and Namibians who joined forces with South African forces against liberation movements in their own countries in the 1980s.
Lawyer Alwyn Griebenow said he was particularly concerned about the condition of one of the men, Francisco Marques, who had tuberculosis and was so emaciated and weak he could not walk without aid.
The alleged ringleader, former British Special Air Services soldier Simon Mann, who met the charter flight at Harare - remained in prison to serve out the rest of his four-year sentence for attempting to illegally purchase weapons from the state arms manufacturing company.
The two pilots of the Boeing 727 were sentenced to 16 months in prison and are scheduled for release in August. Equatorial Guinea has sentenced 24 other suspected mercenaries from European and African nations to lengthy prison terms.
Sir Mark Thatcher, the son of former British prime minister Margaret Thatcher, pleaded guilty last year in a South African court to unwittingly helping to bankroll the coup attempt.
Prosecutors allege Equatorial Guinea's Spanish-based rebel leader Severo Moto offered the group $1.8 million and oil rights to overthrow President Theodoro Obiang Nguema in the former Spanish colony in West Africa.