About 400 worshippers were praying at the mosque in suburban Mirrabooka in the west coast city of Perth when the shooting occurred on Friday evening.
No-one was injured.
Neville Patterson, Western Australia state police inspector, said: "A single shot was fired from a high-powered rifle from outside the mosque through a window on the second level, narrowly missing a couple of worshippers."
The gunman fled from the scene in a green sports utility vehicle.
The bullet lodged in the mosque wall and is being examined by ballistics experts.
Patterson said it was unclear whether the shooting was an attempt to kill or to intimidate Muslims.
Ahdeliah Edries, who was praying at the mosque, said: "I just felt something whizz past my ear and I immediately, instinctively, put my hands up to my ear and then a couple of seconds later there was an incredible bang."
"This is a criminal act, but my question is: where do the criminals get the courage to do this?"
Ameer Ali, head of the Australian Federation of Islamic Councils
Ameer Ali, head of the Australian Federation of Islamic Councils, said that while this was the first shooting in an Australian mosque, others had been ransacked.
He said that worshippers' cars had also been vandalized.
"This is a criminal act, but my question is: where do the criminals get the courage to do this?" Ali said.
"It comes from what our leaders say in public, so I think our leaders share some responsibility for these acts."
Ali declined to identify the leaders whose comments he thought might encourage such attacks.
The attacks come after some politicians have called on Muslim leaders to denounce violence, amid increasing concerns that Middle Eastern imams have preached radical Islam in Australian mosques.
The Australian government has also announced plans to tighten citizenship requirements.
Some Muslims say the rules unfairly picks out their community.
John Howard, the Australian prime minister, has described some segments of the Islamic community as "utterly antagonistic" to Australian society.
Peter Costello, the ruling Liberal Party deputy, recently said Muslims who wanted to live under Islamic law had no place in Australia, insisting all immigrants had to embrace Australian values.
Tensions between Australians and Muslim immigrants have been high since nine Lebanese Muslims gang-raped numerous women in Sydney in 2000.
In December 2005, tensions again rose as Lebanese and Australian youths clashed at Cronulla beach in Sydney after Australian women complained of being harassed by Middle Eastern immigrants.
During several days of rioting, many people were injured and several churches were attacked.