Last year a panel of lawyers and conservationists reported to the Australian and New Zealand governments that Japanese whaling in the Antarctic could be stopped if Japan was held accountable for dumping waste and for undertaking hazardous refuelling at sea.
The Canberra panel said the activity violates the 46-member Antarctic Treaty System, to which Japan belongs.
New Zealand said it will decide in the coming weeks whether to abandon diplomatic efforts and also file a case in the International Court of Justice against Japan.
Japan's foreign ministry said Australia's decision was regrettable while negotiations were going on within the International Whaling Commission (IWC) on disputes over whale hunts.
|Conservation groups have been aggressive in trying to stop Japan's whaling missions EPA]
"We will continue to explain that the scientific whaling that we are conducting is lawful in accordance with Article 8 of the international convention for the regulation of whaling," Hidenobu Sobashima, the foreign ministry's deputy press secretary, said.
"If it goes to the court, we are prepared to explain that."
Sobashima said the issue "shouldn't jeopardise the overall good relations between Japan and Australia".
Echoing the sentiment, Stephen Smith, the Australian foreign minister, said the two countries have agreed to treat the matter as "an independent legal arbitration of a disagreement between friends".
Japan gets around an international ban on commercial whaling by arguing that it harpoons hundreds of whales each year for scientific research.
The IWC is separately trying to resolve long-running disputes over the hunting of whales by several countries despite a 25-year-old moratorium on hunts.
The plan would effectively allow commercial whaling for the first time since the ban, but under strict quotas.
Japan, Norway and Iceland, which harpoon around 2,000 whales annually, argue that many species are abundant enough to continue hunting them.
They are backed by around half the IWC's 88 member nations.
Australia has declared the southern seas a whale sanctuary and has long lobbied for an end to whaling there.
The annual Japanese whale hunt also is a target for protests by conservationists, with vessels of the US-based Sea Shepherd Conservation Society sometimes clashing with the whaling fleet.
On Thursday, Peter Bethune, a Sea Shepherd member from New Zealand, admitted in a Tokyo court to several charges, including trespassing and destruction of property, when he boarded a Japanese whaling ship as part of a protest in February.
He could be jailed up to 15 years if found guilty.