The International Court of Justice (ICJ) in The Hague, also known as the World Court, opened the case on Monday, 13 years after Bosnia sued the rump Yugoslav state from which it seceded in 1992.
The secession triggered a war in which at least 100,000 people were killed.
"The armed violence which hit our country like a man-made tsunami in 1992 ... destroyed the character of Bosnia and Herzegovina and certainly destroyed a substantial part of its non-Serb population," Bosnia's lawyer, Sakib Softic, told the court.
"We are here because the Belgrade authorities have knowingly taken the non-Serbs of Bosnia and Herzegovina on a path to hell, a path littered with dead bodies, broken families, lost youths, lost future, destroyed places of cultural and religious worship."
The hearings at the court, set up after the second world war to mediate in disputes between states, are scheduled to run until 9 May. A ruling is expected by the end of the year.
Bosnia's Muslims and Croats followed Slovenia and Croatia in breaking away from Yugoslavia in April 1992, against the wishes of Bosnian Serbs, who were left as a one-third minority in what had previously been a Yugoslav republic ruled from Belgrade.
Tens of thousands of non-Serbs
were killed by Bosnian Serbs
Backed by the Yugoslav army, the Serbs responded by swiftly
capturing two thirds of Bosnia, besieging Sarajevo and launching "ethnic cleansing" in which tens of thousands of non-Serbs were killed and hundreds of thousands forced out from their homes.
In its application to the court in 1993, when the war was still raging, Bosnia said agents and surrogates of Serbia and Montenegro "killed, murdered, wounded, raped, robbed, tortured, kidnapped, illegally detained and exterminated the citizens of Bosnia-Herzegovina."
While Bosnia's lawyers say their primary objective is to establish that Serbia committed genocide, the country could also seek hefty compensation from its neighbour if it won the case.
Serbia and Montenegro's lawyer, Radoslav Stojanovic, admitted that individuals might have wanted to kill Bosnian Muslims, but said the court could not prove that the state or the Serbian people had intended genocide.
The case has been delayed by a series of counter claims filed by Serbia, which questions the court's jurisdiction.
The UN war crimes tribunal, not far from the ICJ in The Hague, has already determined that the 1995 Serb massacre of 8000 Muslims at Srebrenica, eastern Bosnia, was genocide.
The 1995 massacre of Muslims in
Srebrenica was a turning point
That tribunal is also trying Slobodan Milosevic, the former Yugoslav president, for genocide, crimes against humanity, and war crimes.
Bosnia's lawyers at the World Court will draw on evidence from that and other trials at the tribunal.
Survivors of the Bosnian war demonstrated outside the World Court, displaying a banner with the names of the victims of the Srebrenica massacre under the slogan "Europe's shame".
"We want to show we have not forgotten all the crimes," said one of the protesters, Mustafa Mesanovic. "We want to show all the nations, all the world ... that we Bosnians want justice."
The Bosnian war ended with the 1995 Dayton peace agreement, which divided the country into two highly autonomous regions - a Muslim-Croat federation and a Serb Republic - under a loose umbrella central government.