In his resignation letter, Ljajic said he still had full confidence in his team "and their devotion to reach the goal".
He said he and his team, which was formed in 2006, had "done our best" and had never made a "bigger or more serious effort" to locate Mladic.
Mladic and Goran Hadzic, the Croatian Serb wartime leader, are the only remaining fugitives wanted by the International Criminal Tribunal for Former Yugoslavia (ICTY) and are believed to be in hiding in Serbia.
Mladic has been at large since he was indicted by the Hague-based tribunal in 1995.
He is wanted for his actions in the 1992-95 war in Bosnia, and his alleged role in the 1995 massacre of about 8,000 Muslim men and boys in Srebrenica.
Mladic had lived freely in Serbia under protection from the nationalists before going into hiding a few years ago.
Both Ljajic and Vukcevic had repeatedly said they expected Mladic to be arrested by the end of the year.