The non-binding resolution was proposed by New Zealand and sponsored by other anti-whaling nations led by the US, Britain, Australia, France and South Africa at the annual talks in Anchorage, Alaska.
Russia and Norway voted against it while China abstained.
Chris Carter, New Zealand's conservation minister, said: "It was a clear division showing the different philosophical camps that are here in the IWC.
"It will be bad enough if Japan was just harpooning whales along its own shores, but actually it comes down and harpoons our whales," he said.
Japan called the move a "hate" resolution.
International Whaling Commission (IWC) founded 1946 to regulate and protect whales, which had been hunted to the brink of extinction
Blanket moratorium on commercial whaling introduced 1986
Three IWC members hunt whales in significant numbers: Japan, Norway and Iceland
An exemption to the moratorium allows indigenous communities in places like Greenland and Alaska to hunt whales.
Significant policy changes such as lifting moratorium require a 75 per cent majority vote in IWC
Glenn Inwood, the Japanese delegation spokesman, said: "We had a spirit of compromise at this meeting and the anti-whalers just can't help themselves but to pass hate resolutions."
For the first time in IWC history, more than two dozen countries refused to participate in the vote, saying it was divisive and served no purpose, officials said.
It underscored the increasing division within the 75-member IWC, which has nearly an equal number of pro- and anti-whaling nations, making it difficult for policy changes in the commission that require a three-fourths majority.
Eugene Lapointe, president of the International Wildlife Management Consortium World Conservation Trust, a pro-sustainable use group, said: "It was the wrong resolution, at the wrong time and delivered in the wrong tone.
"It has further undermined the relevance of the IWC."
The IWC, which manages whaling and is in charge of conservation of the creatures, had imposed a moratorium on commercial whaling since 1986 but Japan, according to environmentalists, is exploiting a legal loophole allowing whaling for scientific research.
Japan kills about 1,000 whales a year under its scientific programme then sells the meat.
Japanese whaling fleets head for the Southern Ocean every summer.
"I just think that the Japanese have been very selfish and going against the views of the people of the southern hemisphere," Carter said.
Moratorium overturn bid
Japan last year won a non-binding resolution in favour of commercial whaling, but fell short of the numbers needed to overturn the two-decade moratorium.
|Japan kills about 1,000 whales a year|
under its scientific programme [ICR]
This year, the anti-whaling group has a narrow majority.
Also on Wednesday, Japan made a formal bid at the IWC to overturn the moratorium even though it is expected to fail and worsen a split in the group.
Joji Morishita, senior Japanese delegate, argued that the moratorium had to be lifted to allow Japan's traditional coastal communities to conduct whale hunts.
He said they had the same fundamental right to pursue whaling as natives in the US and Russia, submitting the proposal under the same IWC rules allowing aboriginal subsistence whaling quotas.
The US, Britain, India, France, Australia and Brazil were against the Japanese proposal, to be decided on Thursday, the final day of the four-day talks.