Jawbone with three teeth from the earliest known modern human in EuropeJawbone and teeth reveal humans living at the edge of what was then the habitable world. Photograph: Chris Collins/Natural History Museum, London/Torquay Museum

A fragment of human jaw unearthed in a prehistoric cave in Torquay is the earliest evidence of modern humans in north-west Europe, scientists say.

The tiny piece of upper jaw was excavated from Kents Cave on the town's border in the 1920s but its significance was not fully realised until scientists checked its age with advanced techniques that have only now become available.

The fresh analysis at Oxford University dated the bone and three teeth to a period between 44,200 and 41,500 years ago, when a temporary warm spell lasting perhaps only a thousand years, made Britain habitable.

The age of the remains puts modern humans at the edge of the habitable world at the time and increases the period over which they shared the land with Neanderthals, our close relatives who evolved in Europe and Asia.

Modern humans are known to have interbred with Neanderthals, leaving their mark in the genomes of many people alive today, and are implicated in their demise 30,000 years ago, perhaps by outcompeting them for food and other crucial resources.

The remains are close in age to the first examples of Aurignacian culture, exemplified by a range of artefacts from flint and bone tools to figurines and cave paintings that date from 45,000 to 35,000 years ago.

"We believe this piece of jawbone is the earliest direct evidence we have of modern humans in northwestern Europe, at a site at the very outermost limits of the initial dispersal of our species," said Tom Higham, deputy director of the Oxford radiocarbon accelerator unit, who led the study.

"It confirms the presence of modern humans at the time of the earliest Aurignacian culture, and tells us a great deal about how rapidly our species dispersed across Europe during the last ice age. It also means that early humans must have co-existed with Neanderthals in this part of the world, something which a number of researchers have doubted.

"For many years, people thought Europe was a bit of a backwater, a Neanderthal stronghold almost, but the dating we've done suggests that is not so clearcut," he said.

The early humans who arrived in Torquay were pioneers who ventured into Britain along with other animals and either retreated or were wiped out when temperatures plummeted again at the end of the warm spell.

The bone fragments were originally dated at Oxford in 1989, but the reliability of the dating was called into question when scientists spotted traces of modern glue that had been used to stick the teeth in place.

Too little of the specimen was uncontaminated to re-date the fragments directly, so Higham's team scoured the archives at Torquay museum for animal bones excavated above and below the spot where the jawbone was found.

Having collected the bones of wolf, deer, cave bear and woolly rhinoceros, the team used a new method called bone collagen ultrafiltration to remove modern contamination before dating the samples to between 50,000 and 26,000 years old. The team next used a statistical method to calculate the age of the human jawbone.

The findings are published in the journal Nature alongside another study that bolsters evidence for early human occupation in Europe.

In the second paper, researchers led by Stefano Benazzi at the University of Vienna, re-examined two milk teeth found in a cave in southern Italy in 1964.

The teeth, thought to be Neanderthal, were uncovered at the Grotta del Cavallo next to artefacts described as "Uluzzian", a culture defined by personal ornaments, bone tools and colourants.

Benazzi's team looked at the teeth afresh using digital scanners and concluded that they had the hallmarks of modern humans instead of Neanderthals. The teeth were too small to date directly, but decorative shell beads uncovered in the same archaeological layers as the teeth were found to be 41,000 to 45,000 years old.

The finding "undermines the idea that Neanderthals were engaged in highly symbolic behaviour," Benazzi said. Scientists have uncovered other evidence that suggests Neanderthals made Uluzzian tools and ornaments at sites in France, but at some of these, it is difficult to be sure that the artefacts are the work of Neanderthals.Our ancient human cousins

• Neanderthals

Our close relative, Homo neanderthalensis, emerged in Europe and Asia roughly 130,000 years ago and died out in Europe as recently as 30,000 years ago. They had larger brains than modern humans and there is good evidence they made and used tools. Some sites indicate that these hunter-gatherers buried their dead. Modern humans mated with Neanderthals and their genes live on in non-Africans today.

• Denisovans

A tooth, a toe bone and fingertip found in a Siberian cave that date back to 41,000 years ago are all we have of the Denisovans. DNA analysis reveals that modern humans interbred with them too.