A small, wide-eyed beast with luxuriant orange fur has been identified as a new species more than 100 years after it first went on display in the world's museums.

The discovery brings to an end one of the longest zoological cases of mistaken identity and establishes the "olinguito" (which rhymes with mojito) as the first new carnivore recorded in the western hemisphere for 35 years.

The animal – which has been described as a cross between a teddy bear and a house cat – had been displayed in museums around the globe and exhibited at numerous US zoos for decades without scientists grasping that it had been mislabelled.

One adult female, named Ringerl, was kept at Louisville zoo in the 1960s, but was moved to Tucson zoo, to the Smithsonian's National zoo, and to the Bronx zoo after keepers repeatedly failed in their attempts to breed the animal. The reason for that failure is now clear: it was a different species to the mates on offer.

The true identity of the overlooked beast only emerged after Kristofer Helgen, curator of mammals at the Smithsonian's National Museum of Natural History, launched a 10-year investigation into an obscure group of raccoon-like mammals called olingos. What began with a drawer-full of remains ended with a nighttime trek through the cloud forests of Ecuador, where scientists photographed the creature living in the trees.

"If you look up olingos in a book today, pretty much everyone says we don't know quite how many species there are, what their ranges are, and which are endangered. I set out to resolve all that, I wanted to put olingos on the map," Helgen told the Guardian.

"But in the process of trying to do that, and because we were the first group in generations to look closely at his part of the carnivore family tree, we revealed this incredible and beautiful animal that everyone had overlooked," he said.

The moment of realisation came when Helgen was going through skins and skulls of mammals at the Field Museum in Chicago. "I pulled out a drawer and there were these brilliant, beautiful orange-red pelts with long flowing fur. It was nothing like olingo fur. I then looked at the skulls and the shape was very different. I wondered, 'is this a mammal that's been missed by every other zoologist?' It turns out that it was," he said.

The animal had been mistaken for an olingo because of some broad similarities, but these turned out to be superficial. Helgen's animal was different on almost every measure: it was smaller, much furrier, had a shorter tail, different teeth, and smaller ears. "We are not talking about splitting hairs. If you saw the two animals side by side you would wonder how they could ever be confused," Helgen said.

Convinced they had a new species on their hands, Helgen's team arranged an expedition to the cloud forests of the Andes, where similar creatures had come from. Trekking at night through the dense vegetation, and accompanied by a chorus of frogs and crickets, they spotted other nocturnal beasts in the beams of their headtorches: kinkajous and porcupines.

"Eventually, there it was, an olinguito. We got it in the beam, running around, jumping from tree to tree, but getting close enough so that when it turned and looked into the beam we knew exactly what it was," he said.

The olinguito is a carnivore, but the term has two meanings in biology. The most familiar is an animal that eats meat, but the other is any animal that belongs to the order Carnivora, which includes cats, dogs, tigers, bears and others. They are not all meat eaters, and the olinguito mostly eats fruit.

Working with local museums, the team later extracted DNA from animals on display and confirmed that some were olinguitos, a previously unknown relative of the olingo. They have since confirmed there are at least four sub-species of the animals.

The DNA evidence took the scientists back to the Smithsonian Institution. There they found that scientific databases already contained olinguito DNA that had been wrongly labelled as olingo. It also led them to tissues from a Colombian olinguito held in storage at the museum. They belonged to Ringerl, the unfortunate female that toured US zoos.

"We tracked down Ringerl's keeper and asked why she moved her around so much. She said 'we couldn't get her to breed with any of the olingos.' This animal wasn't fussy, it just wasn't the same species. It would have been impossible. It was a glorious case of mistaken identity," said Helgen.

The name olinguito means small or adorable olingo, but writing in the journal ZooKeys, the team give the animal a formal scientific name too, Bassaricyon neblina. The species name, neblina, means "fog" or "mist" in Spanish, a nod to the cloud forests where the animal lives. But it also means obscured. "That's exactly what the olinguito has been," Helgen said. "Lost in the fog."