Australia has said it could argue that Japan's whaling is illegal before the International Court of Justice at The Hague or the International Tribunal for the Law of the Sea in Hamburg, Germany.
Australia, which has long taken a staunch anti-whaling stance, has issued similar ultimatum before but has not followed up on the threats and warnings.
Rudd's threat came on the eve of a visit by Katsuya Okada, the Japanese foreign minister, to Australia, and whaling is expected to be a key topic of conversation at meetings over the weekend.
"If [diplomacy] fails, then we will initiate that court action before the commencement of the whaling season in November 2010"
Speaking to the Seven Network Rudd said Australia would "work with the Japanese to reduce, through negotiation, their current catch to zero".
"If that fails… and I'm saying this very bluntly and very clearly on your programme today… if that fails, then we will initiate that court action before the commencement of the whaling season in November 2010," he said.
"That's the bottom line and we're very clear to the Japanese that's what we intend to do."
Japan's centre-left government has maintained its support for whaling, which it says has deep cultural significance for the Japanese people, since coming to power in September.
Okada is scheduled to meet Rudd and John Faulkner, the Australian defence minister, on Saturday before holding talks with his Australian counterpart, Stephen Smith, on Sunday.
Okada said ahead of the trip that he plans to "discuss global issues such as climate change, nuclear disarmament and non-proliferation as well as bilateral issues".
He added: "I also expect to discuss whaling."
"We remain in the position that our research whaling mission is legal," Okada told a news conference in Tokyo.
"I would like to have a discussion with Prime Minister Rudd to help him to understand our position well."
Japan hunts hundreds of mostly minke whales, which are not an endangered species, in the Antarctic each year under its whaling research programme.
|Critics say the sale of whale meat is the real reason behind Japan's programme [EPA]
Although that is an exception to the International Whaling Commission's 1986 ban on commercial whaling, critics say the real reason for the hunts is to sell whale meat that is not used for study.
On Wednesday, a group of conservationists clashed with Japanese whalers in the Antarctic Ocean, the latest in a string of increasingly aggressive confrontations between US-based activist group Sea Shepherd and the whaling fleet.
Sea Shepherd activists threw bottles of butyric acid at Japanese whalers and blasted their ship with paint, while the Japanese returned fire with water cannons.
No one was injured, but Japan condemned the conservationists' actions as dangerous and violent.