The carnivorous mammal, slightly larger than a domestic cat with dark red fur and a long bushy tail, was photographed twice by an automated camera at night in 2003 on the Indonesian side of the island, the WWF said on Tuesday.
Neither Bornean wildlife experts nor locals well acquainted with the area recognised the animal, the group said.
The animal, which has very small ears and large hind legs, was spotted in the Kayan Mentarang national park in the mountainous jungles of Kalimantan, where vast tracts of rainforest still remain.
"Most were convinced it was a new species of carnivore," WWF said, adding that researchers were hoping to set traps to catch a live specimen.
WWF ecologist Stephan Wulffraat said a live capture of the animal was required to confirm it was a new species.
Between cat and fox
The animal appeared to be a cross "between a cat and a fox" and may live in trees during the day, coming down at night, he said.
The group said it was extremely rare these days to discover a new mammal species of this size, particularly a carnivore.
An automated camera captured
the animal's image twice
If confirmed, it would be the first time in more than a century that a new carnivore has been discovered on the island, which lies between the Indian and Pacific Oceans.
WWF warned that plans to create the world's largest palm oil plantation in Kalimantan, along the border with Malaysia's Sarawak and Sabah states, threatened further new discoveries.
The scheme, funded by the China Development Bank, is expected to cover an area of 1.8 million hectares.
Environmental watchdogs have criticised the plan, arguing that the jungle soil in the area is infertile and that the elevation is unsuitable for palm oil.
Indonesia is losing at least 2.8 million hectares of its forests every year to illegal logging alone.
An artist's rendering of the
Bornean red carnivore
The Forestry Ministry's director for protected areas, Banjar Laban, said the potential discovery of a new mammal emphasised the urgent need to protect the biodiversity of Borneo's forests.
"If it turns out to be truly a new mammalian species, this should really become a national pride, something that the entire nation should be proud of and work to preserve," he said.
In the protected Kayan Mentarang forest, 361 new species - plants, insects, fish and other animals - were discovered between 1994 and 2004, he said.
Rapid deforestation has had devastating environmental consequences for Indonesia and the Southeast Asian region, causing floods and landslides and shrouding nearby countries with haze from illegal fires set to clear land.