Wednesday 22 August 2012 04.00 EDT
First published on Wednesday 22 August 2012 04.00 EDT

Dan is a detective sergeant, and a covert internet investigator with the Metropolitan police. At the Met's paedophile and high tech crime unit, we sit in front of his laptop as he navigates to an internet chatroom. He's already set me up with a username — louiseis14 — and clicks the button that tells users I'm online.

Within 10 seconds, two men whose profile names suggest they're in their 30s and 40s want to talk to me. Dan selects one at random — lets call him 'anonman_37' — and starts typing a response.

As the conversation goes back and forth, notifications from men inviting me to get in touch appear so fast that the list runs off the screen. We've been online for less than three minutes.

Dan logs out quickly: he doesn't want to get into a conversation which might lead to a crime being committed. This would require his team to mobilise an instant response, something officers here do regularly, but not typically with a journalist watching proceedings.

The internet has provided new opportunities for paedophiles to prey on children, and the demonstration Dan's just given of just how many are waiting online to engage with young girls is alarming. "It's either the biggest library in the world or the biggest porn shop in the world," he says.

As one of 60 investigators working specifically on online paedophile crime at the Met, Dan knows better than most the ease with which (mostly) men can access and groom minors on the internet. He's also engaged in intimate online interactions with some of the most dangerous child sex offenders operating in the London area today.

Since the Sex Offences Act (2002), which criminalised grooming via the internet, there has been a growing realisation in the Met that it would have to change the way it operates: to successfully pursue and prosecute the growing communities of paedophiles who commit sex crimes online, it would have to penetrate the various 'meeting places' made possible by the explosion in social networking sites. Crimes committed on these sites include causing a child to engage in sexual activity, engaging in sexual activity in the presence of a child online, arranging or facilitating the commission of a child sexual offence, and meeting a child following sexual grooming online.

Waiting until children are taken into care to investigate sex crimes is no longer an option when offenders are targeting minors in such an organised manner. Police now use proactive targeting using intelligence from a range of sources. To get better at building cases against the most dangerous online perpetrators, the unit's officers, overseen by detective superintendent Reg Hooke, have made it their mission to gain a detailed understanding of how offenders operate within support networks of fellow paedophiles.

To help them, Elena Martellozzo, a criminologist at Middlesex University, has been given close access to the work of the unit and the chance to observe the way that paedophiles interact online with officers assuming the identities of both vulnerable children and predatory paedophiles.

Her analysis of a single recent investigation that led to 27 arrests of men who set up, and then travelled to meetings with "children" forms the basis of a new training course that, she says, aims to extend undercover officers' understanding of the various modus operandi used by sex offenders who primarily target young children on the net.

Called "Dancing with the Devil, investigating online child abusers", the training examines a range of perpetrator profiles and looks at how well the police currently distinguish between different types of online predator.

Martellozzo looked at their profiles "from super confident and open about their tastes to very cautious" in their online descriptions — the hope is that her training will offer police an increasingly sophisticated understanding of paedophiles' online behaviours and allow covert investigators to more quickly establish who are the fantasists and who the most dangerous predators.

This training, Hooke says, is also helping his officers in policing such crimes.

He knows full well that proactively targeting and engaging in conversations with online sex offenders holds some risks. There are moments when the "legally audacious culture" he is developing in this unit — required if officers are to engage believably in online conversations — takes his team very close to the legal line. Clearly, they are not allowed to act as agent provocateurs or encourage someone to commit a crime that they otherwise wouldn't.

It takes considerable support from management before officers feel able to throw off the risk averse culture that Hooke acknowledges can hold investigations back. But the approach is working: weekly, if not more often, he confirms, people are being arrested, prosecuted and sentenced for online sex crimes against minors in chatrooms, on social networking sites and via instant messenger. Prosecutions have been supported by chatlogs of officers acting as vulnerable children, and careful recording of all information proffered willingly by the offenders themselves.

The men — and it is mostly men — who offend in this way come from all walks of life, Martellozzo explains in her book Online Child Sexual Abuse: Grooming, Policing and Child Protection in a Multi-Media World. Her analysis of those arrested in the operation described above showed that half were married or in relationships, most had no previous convictions for sex offences, and the vast majority had child abuse imagery on their computer (half had level five imagery — the most extreme) sufficient to have been charged on that basis alone. The majority had their own children, who did not disclose abuse when interviewed.

In their interactions with undercover investigators, half of the men arrested texted the 'victim'. Two-thirds exposed themselves on a webcam or in a photo, and the same proportion actually set up a rendezvous and travelled to meet the 'child'.

When caught, explains Martellozzo, "they say things like 'it was just a fantasy' but you find them with lubricants and toys — what I found very interesting was to witness the forcefulness of their online interactions, [and contrast that] with their subsequent denials in interviews."

Protecting children online is expensive both in police hours and, in a fast-moving digital world, in the regular upgrades to the kit required to stay credible to the paedophiles they interact with. "When budgets are tight, then this proactive covert type of operation is in the balance," says Hooke. His bosses, however, have indicated that they remain committed to funding the unit: Hooke is clear that the long-term value to society will be worth the money.

"Take an abused child and look at the long-term cost to health, education, the justice system and lost work opportunities for the rest of their lives," he says. "In economic terms, this work makes complete sense."

• Names of some officers have been changed to protect their identities