Serbia's state-run Tanjug news agency said that Belgrade had filed a "counter-complaint against the Republic of Croatia for genocide committed against Serbs during the 1991-1995 war".
The lawsuit apparently focuses on the closing period of the war in 1995 when an estimated 200,000 Serbs left Croatia in the face of a military operation to retake territory from Serb fighters who controlled one-third of the country.
Vuk Jeremic, Serbia's foreign minister, said that his country had done all it could to avoid filing the suit, but had acted after Croatia had refused to withdraw its own.
"We had no choice," he said.
"We are aware that we are now entering a new phase in relations with Croatia, but we still want to co-operate with that country in the process of European integration, just as we want peace and stability in the western Balkans."
The two neighbouring countries, which were part of Yugoslavia before the wars of the 1990s, have sought to improve their relations in recent years.
But Vojin Dimitrijevic, a Serbian analyst, told the Serbian newspaper Danas that the suit and countersuit could cause problems for the two countries as they seek European Union membership.
"Both lawsuits are a waste of time and energy, and the worst thing is that they will harm the improved relations between the two countries and make life difficult for Serbs living in Croatia," he said.
Zarko Puhovski, a professor of political philosophy at Zagreb University in Croatia, said that neither lawsuit had much chance of success.
"Those are political suits, and there is no doubt among experts that neither side has any chance of winning," Puhovski told Danas.
He also said the public in Croatia was largely uninterested in the lawsuit "because people are more occupied with economic problems and surviving".
The war in Croatia broke out in 1991 after Croatia split from the Yugoslav federation.
Croatia wants the ICJ to order Serbia to pay compensation for the war, to help trace missing Croatians, and to return cultural items taken during the conflict.
In a similar lawsuit by Bosnia against Serbia in February 2007, the ICJ cleared Belgrade of committing genocide itself but said it had not done enough to stop the actions of the Bosnian Serbs.
The ICJ, based in The Hague, is the highest legal authority of the United Nations and was inaugurated to resolve disputes between states.