The Australian-declared sanctuary – which Japan does not recognise - generally extends to 370 km from the coast of Australia's Antarctic territory, according to the Australian government.
The area forms a large part of the regular hunting grounds used by Japanese whalers.
Kyodo Senpaku Kaisha owns ships currently hunting in Antarctic waters and effectively represents the Japanese whaling industry.
The hearing was derailed in 2005 when the Australian government intervened fearing it could spark a diplomatic row with Japan, arguing the matter was best left to governments.
But the federal court ordered the proceedings to resume in 2006.
Passing judgement on Tuesday federal court Judge Jim Allsop ruled the whaling firm had "killed, injured, taken and interfered with Antarctic minke whales and fin whales and injured, taken and interfered with humpback whales in the Australian Whale Sanctuary in contravention of… the Environment Protection and Biodiversity Conservation (EPBC) Act."
"Our intention to continue to have an overall, holistic and fair-dinkum approach to opposing Japanese so-called scientific whaling is absolutely clear"
Peter Garrett, Australian environment minister
He ordered that the firm "be restrained" from future hunts, but conceded there was little chance his ruling could be enforced.
"Unless the respondent's vessels enter Australia, thus exposing themselves to possible arrest or seizure, the applicant acknowledges that there is no practical mechanism by which orders of this court can be enforced," Allsop said.
Japan, like most countries, does not recognize Australia's territorial claim on Antarctica or its surrounding waters, and says the Australians government has no authority to enforce its domestic laws on the high seas.
Peter Garrett, Australia's environment minister, said he would be looking carefully at the judgement to see if it could add weight to an Australian government challenge to whaling in an international tribunal.
"The Commonwealth wasn't a party to this case but our intention to continue to have an overall, holistic and fair-dinkum approach to opposing Japanese so-called scientific whaling is absolutely clear," he told reporters.
Commercial whaling has been banned for more than two decades but a loophole allows Japanese whalers to hunt a limited number of whales for "research purposes".
It then sells the harvested whale meat to raise funds for further research missions.
However, environmentalists argue that research into whales can be carried out without killing them.
On Monday environmental group Greenpeace said they had succeeded in disrupting the current whale hunt, chasing the main whale processing ship from hunting grounds near Antarctica.
The Japanese whaling fleet plans to hunt around 1,000 mainly minke whales this season.