The 25-million-year-old fossil is of an early type of baleen whale, a group including modern humpback whales, minke whales and blue whales that feed via baleen, comb-like plates in their mouths that filter plankton from sea water.
"This bizarre new baleen whale did not even have baleen," said Erich Fitzgerald, of Monash University in Australia, of the small whale, that was probably up to 3.5m (11.5ft) long.
"It had teeth and was a powerful predator that captured large fish, perhaps sharks, maybe even other whales," he said.
"Some of the early baleen whales weren't gentle giants."
Most scientists have believed that baleen whales quickly evolved baleen for feeding on tiny fish and plankton after breaking from an ancestor shared with toothed whales almost 40 million years ago.
Modern toothed whales include dolphins, killer whales and sperm whales - the species made famous as the bane of Captain Ahab in Herman Melville's novel Moby Dick.
The animal had long teeth and
large eyes [pic: Museum Victoria]
"This rewrites the picture of baleen whale evolution," Fitzgerald said.
The fossil was found near Jan Juc, a town in Victoria, south-eastern Australia, and dubbed "Janjucetus".
Its sharp teeth were about three centimetres long and it had large eyes, apparently suited for hunting, according to Fitzgerald's report, published in the journal Proceedings of the Royal Society B.
Blue whales, which can exceed 150 tonnes and grow longer than 30m, are the largest creatures ever to inhabit the earth - bigger than any dinosaur.
Whales evolved from land mammals, where their closest relative is the hippopotamus.