Isam Basrawi, a lawyer for the defendants, said the three fired one of their lawyers after he presented to the judge a defence plea asking for their release.
The defendants wanted the document back because it should have been presented at an open session, he said on Monday.
The judge refused, so the defendants fired their lawyer. At that point, the session apparently ended.
"They rejected their lawyer's move because the hearing was not open," said Basrawi, who said he was barred by the justice ministry from representing the three.
He said one of the presiding judges had promised the lawyers Monday's session would be open, like the first one, but fewer than 10 relatives were allowed into the courtroom.
An open trial would be remarkable in a nation where trials are normally conducted privately.
Police have moved quickly to
break up political protests
Reformists had hoped the public first session was part of a slow, limited government trend towards more openness.
Two hundred people watched the initial hearing.
"This is an unfortunate setback," said Fawziya al-Uyuni, al-Dimini's wife, who could not attend.
"The judge had promised an open trial. He's gone back on that promise," added al-Uyuni, who had travelled 400km from her home in the eastern town of Dhahran.
The three defendants, Matruk al-Falih, Ali al-Dimini and Abd Allah al-Hamid, are the last remaining detainees to be dealt with among 13 reformers arrested in March.
They had openly criticised the kingdom's government and slow pace of reform.
Some of the 13 had signed a letter to Crown Prince Abd Allah calling for political, economic and social reforms, including parliamentary elections.
The defendants are charged with sowing dissent, creating political instability, printing political leaflets and using the media to incite people against the government.
Relatives say they are being held in a government villa, have access to newspapers and television and are allowed family visits.
Family members milled around the hall outside the courtroom for two hours waiting for permission to enter.
Female relatives waited with their children behind wooden screens.
"We were initially happy that they would have an open trial"
Isa al-Hamid, brother of accused
Soldiers closed the corridor leading to the room.
"This is a big disappointment," said Isa al-Hamid, Abd Allah al-Hamid's brother. "We were initially happy that they would have an open trial."
When word emerged that the two men had been led out of the courthouse, Ali al-Qithami, an activist, led the group of relatives and supporters down the stairs.
Camera phones banned
After the session ended, one of the lawyers, Abd al-Rahman al-Lahim, was led away by court security for bringing a camera-equipped mobile phone to the proceedings, his colleagues said.
Such phones have been banned by the kingdom and authorities wanted to be sure he was not taking pictures during the hearing.
The judge did not set a date for the next hearing.
The hearing on Monday was the defendant's third since 9 August. The hearing on 23 August was postponed after 400 supporters, in a rare show of public dissent, demonstrated for access and then interrupted the hearing with shouts for reforms once allowed into court.
The detentions caused tension between Riyadh and Washington after the US state department condemned them as "inconsistent with the kind of forward progress that reform-minded people are looking for".
The Saudi government says it is
disappointed by the US response
The Saudi foreign ministry replied it was "disappointed" by the US reaction.
The Saudi royal family has absolute power, and Saudis cannot hold public gatherings to discuss political or social issues.
However, fear of unrest brought home for Saudis after the 12 May bombings that killed 35 people, including nine alleged Saudi attackers, has initiated unprecedented public debate.
Last year, police were quick to arrest participants in a Riyadh protest and stifled a follow-up demonstration in Jeddah, both of which were organised by a London-based Saudi dissident group.
The Movement for Islamic Reform in Arabia had organised protests to demand the release of some of the hundreds of people arrested in the aftermath of the 12 May 2003 bombings in Riyadh.
It maintains that the "security scare" and "terrorism" are being used as excuses to round up political opponents and jail innocent people.