Babic, who was sentenced to 13 years in prison for crimes against humanity, was found dead on Sunday evening in his cell at the UN detention centre in Scheveningen, a suburb of The Hague, said a tribunal statement on Monday.
Babic was a ranking Croatian Serb leader when the Serb minority revolted after Croatia broke away from Yugoslavia in 1991, setting off a war that lasted until 1995.
He later was a pivotal witness at the ongoing war crimes trial of Slobodan Milosevic, the former Yugoslav president.
Babic's family was informed on Sunday after the chief medical officer of the centre confirmed that the cause of death was suicide. A tribunal judge ordered an inquiry. The tribunal did not say how Babic killed himself.
It is the second time a Balkan detainee has committed suicide. The first was Slavko Dokmanovic, another Croatian Serb leader, in 1998.
Babic, 50, pleaded guilty in 2004 to a single charge of inflaming an ethnic-cleansing campaign that killed hundreds of Croats and expelled tens of thousands in the self-proclaimed Croatian Republic of Krajina.
The plea was part of a deal in which prosecutors dropped four other charges of murder, cruelty and the wanton destruction of villages during the war in Croatia.
"I stand before this tribunal with a deep sense of shame and remorse. I allowed myself to participate in the persecution of the worst kind against people only because they were Croats, not Serbs"
Milan Babic, former Croatian Serb leader in 2004
His appeal against the lengthy sentence was rejected last July and he was transferred to another prison to serve his sentence.
He returned to The Hague last month to testify against Milan Martic, who became the leader in Krajina after Babic broke with Milosevic, his former mentor.
Babic was accused, along with Milosevic, of being part of a conspiracy, or "joint criminal enterprise," to clear about one-third of Croatia of non-Serbs and incorporate that area into an ethnically pure Serbian state.
Unlike Milosevic - who rejects the legitimacy of the UN court and was extradited to The Hague by the Serbian authorities - Babic voluntarily surrendered to face the charges.
In his plea agreement, Babic said he knew nothing of the murders of non-Serbs until many years later and had no knowledge at the time of the extent of the crimes being committed by Serb forces. Before sentencing, he apologised to the Croatian people.
"I stand before this tribunal with a deep sense of shame and remorse. I allowed myself to participate in the persecution of the worst kind against people only because they were Croats, not Serbs," he said. "I ask my brother Croats to forgive us, their brother Serbs."
War crimes tribunal
Babic's testimony in 2002 against Milosevic provided a dramatic highlight to a trial that has gone on for more than four years.
Though Milosevic denied involvement in the Serb uprising in Croatia, Babic told the court he in fact played a key political and military role behind the scenes and had "pulled the strings" of the Krajina Serbs.
Babic testified against Milosevic,
the former Yugoslav president
The two men hurled accusations at each other while the three UN judges looked on.
"You dragged the Serb people into war," Babic said, charging that Milosevic failed to protect the Serbs. "You brought shame on the Serbs."