"In line with the law, they must have an address so that justice can be carried out," Kaim said.
"They can go back to their homes [in capital Tripoli] or to a hotel but they must leave the embassy."
Kaim said that his government had asked the Swiss embassy to make the men available to authorities for trial dates and other court procedures.
The pair were arrested two days after Hannibal Gadhafi, the son of Moammar Gadhafi, the Libyan leader, and his wife were detained for two days for allegedly physically abusing their servants in a Geneva hotel.
Goeldi, a senior manager at ABB, the engineering giant, and Hamdani, who works for a small construction firm, were originally refused exit visas from Libya and charged with immigration offences.
They were later released on bail and handed to the Swiss embassy, raising hopes that they would be freed.
Bern and Tripoli struck a deal in August that would have seen them return home.
But in mid-September the pair were invited for a medical check up outside the embassy and taken away by Libyan officials, before being returned to the embassy in early November.
Kaim said Bern must "stop politicising this case in order not to jeopardise the situation" of the Swiss businessmen.
"I don't understand why the Swiss government and press insist on linking a court case to the diplomatic row."
Gadhafi was released from Swiss custody when the complaints were dropped after the two servants received compensation from an undisclosed source.
The initial arrest also led to Libya recalling some of its diplomats from Switzerland and freezing some of its business ties with the country.