No-one was injured in the attack, which occured in the Mostar suburb of Jesenice during the Muslim fasting month of Ramadan, said Lejla Trivun, Mostar police spokeswoman.
Trivun said the damage was caused by a 60mm Zolj (anti-tank rocket projectile), which was fired at the mosque around 4am local time (0200 GMT).
National television showed footage of the mosque with broken windows and damages to the facade.
It is the most serious incident in several years involving this type of weapon in the post-war Balkans country.
Salem Dedovic, a Mostar imam, told national television that the "barbarian act has upset spirituality and peace among Muslims during the month of Ramadan".
Ljubo Beslic, Mostar's mayor and a Croat, immediately condemned the attack, calling on citizens "to remain calm and help police in identifying the perpetrators".
National tensions remain high in Bosnia almost 11 years after the brutal 1992-95 war that pitted Bosnian Croats, Muslims and Serbs against each other. Mostar remains divided between Muslims and Croats.
It is believed that many Bosnians are still hiding some illegal wartime weapons.