Boban Simsic is charged with crimes against humanity and persecution of Bosnian Muslims in 1992 in the eastern town of Visegrad when he was a member of a Serb paramilitary unit.
President of the court Medizda Kreso labelled the opening of the first trial as a "historic moment for Bosnia-Hercegovina".
Simsic, 38, surrendered in January to European Union peacekeepers, who handed him over to local authorities.
His trial is to continue on 13 October, when the first witnesses are to testify.
Simsic's unit was under the command of Milan Lukic, charged by the UN tribunal at The Hague for war crimes in Visegrad, and arrested in August in Buenos Aires.
Lukic and his cousin Sredoje Lukic, who surrendered to Bosnian Serb authorities earlier on Wednesday, are charged by the UN court with having created and led the White Eagles.
The group killed nearly 150 Muslim civilians including women and children in the eastern town of Visegrad between May 1992 and October 1994.
Easing UN's burden
The Bosnian court, expected to ease the burden of the UN war crimes court at The Hague by taking some of its caseload, was inaugurated in March.
It is staffed mainly by international judges and prosecutors, with local legal officers set to gradually take over.
Bosnian courts have already tried some war crimes cases after receiving approval from the International Criminal Tribunal for the former Yugoslavia (ICTY).
The ICTY will wind up its work in 2010.
With some 50 suspects still awaiting trial, it will have to transfer several cases to courts in the Balkans if it is to meet its target.
Bosnia's 1992-95 war claimed more than 200,000 lives and saw the worst atrocities in Europe since the second world war, including the 1995 massacre of some 8000 Muslim males by Serb forces in Srebrenica.