It was shortly after 10am, in a corner at a primary school near Nottingham, that a police agent using the codename UCO 133 began whispering into a microphone hidden in his watch.

Mark Kennedy was a long-haired, tattoo-covered undercover police officer who had been living for six years as an environmental activist. But the covert agent with a long-term activist girlfriend was about to set in train a chain of events that would result in one of the most intriguing scandals in policing history.

"I'm an authorised police officer engaged in Operation Pegasus," Kennedy hissed into his £7,000 Casio G-Shock watch, equipped with a hidden microchip. "This weekend, Easter weekend, I am together with a group of activists that are planning to disrupt Ratcliffe-on-Soar power station. Shortly gonna go … and record briefings that subsequently take place throughout the day. So I shall now switch this device off."

He snatched a look at his wrist and read out the time. At that point – 10.06am on 12 April 2009 – one of the British constabulary's most closely guarded secrets remained intact; Kennedy, perhaps the most successful in a fleet of agents sent to live deep undercover among political activists, had maintained his cover.

More importantly, virtually nothing was known about the secretive police units which, for four decades, had been surreptitiously disrupting the activities of political campaign groups.

But now a series of revelations concerning a network of undercover agents has become a growing crisis for police.

At the centre of the latest controversy is a set of documents, obtained by the Guardian and the BBC's Newsnight, indicating that another police spy, Jim Boyling, who lived undercover among the environmental group Reclaim the Streets, concealed his identity in a criminal trial, giving false evidence under oath about his real name.

The accusation that police deliberately subverted the judicial process, and at worst sanctioned perjury, prompted outrage among lawyers and parts of the judiciary and led to the last-minute postponement of a major report into undercover policing of protests by the newly appointed commissioner of the Metropolitan police, Bernard Hogan-Howe.

Now questions are being asked about the judgment of Britain's most senior police officer, whose report – conducted in his prior role with the policing inspectorate – is being reviewed. Lord Macdonald, the former director of public prosecutions, described the court deception as a monumental misjudgment, saying police had "crossed the line". There are mounting calls for a full public inquiry.

The truth behind the police spies began to unravel late last year when activist former friends of Kennedy revealed his police background on the website Indymedia.

Two months later – in January this year – the Guardian published the first revelation in its long-running investigation into the undercover policing of protests, revealing how Kennedy, after leaving the Met, returned to his activist friends, expressed sympathy with their cause and attempted to continue living under his fictional identity, Mark Stone.

In the last 10 months, the Guardian has detailed the covert deployments of six undercover police officers. In addition to Kennedy and Boyling, police officers using the fake identities Mark Jacobs, Lynn Watson and Pete Black have been exposed. This week Bob Lambert, a well-known academic, was unmasked as a former spymaster who spent years deep undercover.

Writing in the Guardian , Lambert acknowledges police should learn from mistakes, but defends the work of undercover police officers in "countering political violence and intimidation".

Lambert, who later ran special branch's Muslim contact group, which was tasked with building relations with London's Muslim organisations, said he was not involved in any surveillance at that stage of his career. Boyling also went on to work for the same unit. "I did not recruit one Muslim Londoner as an informant, nor did I spy on them," Lambert said. "They were partners of police and many acted bravely in support of public safety and protection of fellow citizens."

A seventh undercover officer, Simon Wellings, was exposed by Newsnight in March.

All seven spies shared similar modi operandi: they appeared out of nowhere, often had access to vehicles and showed an unflinching willingness to help run the logistics of protest organisation. Unlike undercover officers who penetrate serious criminal gangs, typically for no more than a few weeks or months, agents deployed in protest organisations are authorised to spend years living double lives as campaigners. Only rarely have they been asked to gather evidence for prosecutions; usually, their mandate is to gather intelligence on activists while quietly disrupting their campaigns.

Most of the undercover police officers identified by the Guardian and Newsnight have also had sexual relationships with their targets, in some cases developing long-term relationships.

Some activists argue this has been the most disturbing element of the controversy, equating the operation to state-sanctioned sex abuse. They point to the anger, betrayal and psychological trauma suffered by some of the women who have spoken out about their relationships with men who later turned out to be police spies.

Senior officers have claimed sexual relations were never condoned or known about by the top ranks – a finding Hogan-Howe was expected to endorse in his report. However, the mounting evidence suggests otherwise.

Kennedy said he could not "sneeze" without his handlers knowing about his activities, and gave every indication they knew about the methods he used to gain the trust of activists, including his sexual liaisons. Black has said it was "part of the job" for fellow agents to use "the tool of sex" to maintain their cover and glean intelligence.

Together, these seven agents, and dozens more, have infiltrated a series of groups from across the political spectrum, including groups such as Stop the War, Youth Against Racism, Earth First, and Climate Camp. They have been regularly spying on activists at major demonstrations surrounding summits such as the G8 and G20, as well as local protests such as a campaign to protect Titnore Woods in West Sussex.

However, it was Kennedy's operation to prevent 112 activists from breaking into the Nottinghamshire power station in 2009 that placed the long-running operation under the spotlight.

Late last year, prosecutors refused to admit that the environmental campaigners had been infiltrated by an undercover police officer. The secret recordings made on Kennedy's Casio watch – which would have exonerated the activists if disclosed during their trials – were suppressed. An inquiry by Sir Christopher Rose, the former surveillance commissioner, is investigating claims made by police that their colleagues in the Crown Prosecution Service suppressed the recordings.

Transcripts of those recordings have now been obtained by the Guardian, along with other police materials relating to Kennedy's deployment marked "restricted" and "confidential".

They shed light on the extent of surveillance undertaken to keep tabs on a group of environmental campaigners. They reveal the minute details about the activities of campaigners being relayed by Kennedy, from discussions about football teams to types of biscuits eaten at a planning meeting.

In one document, marked "secret", police chiefs lay out what they believed to be the legal justification for Kennedy's surveillance operation, stating that the environmental campaigners could cause "severe economic loss to the United Kingdom" and an "adverse effect on the public's feeling of safety and security".

Those police claims, along with the broader suggestion that environmental activists threaten the national infrastructure of the UK, have been repeatedly challenged in court. All 26 activists police wanted to prosecute for conspiring to trespass at the Nottinghamshire power station either had their trials abandoned or their convictions quashed following the Kennedy controversy.

Sentencing 20 of the activists in January, a judge at Nottingham crown court said he accepted they had intended a peaceful protest and had the "highest possible motives", describing the group as "honest, sincere, conscientious, intelligent, committed, dedicated, caring".

When their convictions were quashed in July, three court of appeal judges, who included the lord chief justice, said "elementary principles" of the fair trial process were ignored when prosecutors did not disclose the secret recordings to activists' lawyers. In a damning ruling, the judges said they shared the "great deal of justifiable public disquiet", found that Kennedy's operation had been partly unlawful, and even proffered the suggestion he had arguably been acting as an agent provocateur.

What the judges did not mention – but is increasingly becoming clear – was that Kennedy was not a lone operator, but the latest in a long line of undercover police officers who have been spying on activists as part of a classified operation dating back four decades.

Previously known as the special demonstration squad, which operated under the command of the Metropolitan police's special branch, the undercover unit was first conceived as a tool to combat the anti-Vietnam protests at Grosvenor Square in 1968.

The infrastructure of long-term police surveillance of leftwing and far-right campaign groups has remained in place ever since – and continues today. What was previously conceived as a secret plan to disrupt the activities of "subversives" was, more than a decade ago, reinvented under the leadership of the Association of Chief Police Officers as part of a new drive to combat "domestic extremists". The secretive body that controls the spies, the National Public Order Intelligence Unit, was recently returned to the command of the Met.

It now falls to Hogan-Howe to grapple with the fallout of the latest controversy over Boyling, who has been placed on restricted duties and subject to a disciplinary inquiry since January, when it emerged he married an activist he met while undercover and fathered two children with her. That inquiry, which is investigating claims by Boyling's ex-wife that he encouraged her to change her name by deed poll to conceal their relationship from his superiors, has yet to conclude.

It is now likely to be overshadowed by the accusation that he lied about his real identity under oath. Details of his false evidence were revealed on Wednesday.

Besides prompting outrage among lawyers, the accusation that police subverted the judicial process appears to have shaken senior police officers. Within hours, Her Majesty's Inspectorate of Constabulary (HMIC) cancelled its planned publication of the report on Hogan-Howe's review.

The Hogan-Howe report had been expected to ignore advice from other senior police officers, who argued that the unfolding scandal in undercover policing revealed the need for a more robust system of independent oversight. HMIC said it would now seek further details about Boyling's alleged false evidence under oath before reviewing its report. However, what is unclear is how much information – if any – the Met disclosed to the inspectorate about Boyling, his marriage to an activist and his evidence under oath.

A draft of the HMIC report circulated over the summer, as Hogan-Howe believed he was nearing his conclusions, is not believed to have contained any reference to Boyling at all.

Jenny Jones, a Green party member of the London assembly who sits on the Metropolitan police authority, will be questioning Hogan-Howe at an MPA meeting next week. She said: "I will be pressing him to explain how we can stop such mistakes being made again and how we can bring some accountability to a police service which has been given almost carte blanche to spy on its own citizens."