"There is a real possibility we will review at a fundamental level our role in the IWC and this would include withdrawing, setting up a new organization," said Akira Nakamae, Japan's top delegate after the meeting ended.
He said Tokyo would also consider defying the two-decade ban and "unilaterally" carry out whaling along its exclusive economic zone.
"This hypocrisy leads us to seriously question the nature by which Japan will continue participating in this forum"
Joji Morishita, Japan's delegate
Citing what it said was centuries culture, Japan had for coastal communities with a long history of whaling to be allowed to hunt limited numbers of minke whales.
It had sought the same consideration as that endorsed by the IWN earlier this week for aboriginal whaling.
But the fierce opposition from anti-whaling nations forced the Japanese delegation to scrap the idea without calling for a formal vote.
That led Nakamae to accuse the IWC of "double standards" saying Japan and other pro-whaling nations could comprehend why Japan was treated differently from other groups with long traditions of whaling.
"This hypocrisy leads us to seriously question the nature by which Japan will continue participating in this forum," said Joji Morishita, Japan's deputy whaling commissioner, expressing the frustration of the Japanese side.
International Whaling Commission (IWC) founded 1946 to regulate and protect whales, which had been hunted to the brink of extinction
Blanket moratorium on commercial whaling introduced 1986
Three IWC members hunt whales in significant numbers: Japan, Norway and Iceland
An exemption to the moratorium allows indigenous communities in places like Greenland and Alaska to hunt whales.
Significant policy changes such as lifting moratorium require a 75 per cent majority vote in IWC
Japan's threat to leave the IWC brought an end to a meeting that was, at times, both testy and conciliatory.
Japan had earlier suggested a compromise on a controversial plan to hunt 50 humpback whales next year under its scientific whaling program in exchange for support for its coastal whaling proposal.
But Australia and its anti-whaling allies rejected the deal, saying Japan was attempting to hold the endangered humpback whales "hostage".
Malcolm Turnbull, the Australian environment minister, said passing the coastal whaling proposal would have set an irreversible precedent.
"The minute you open the door to commercial whaling, how do you shut it again? That is the problem," he said.
Earlier on Thursday, the IWC passed a proposal to let Greenland expand indigenous whale hunting after the Danish territory broke a deadlock by agreeing not to start catching humpback whales.