In a unique experiment, The Guardian published online the full manuscript of its major investigation into the climate science emails stolen from the University of East Anglia, which revealed apparent attempts to cover up flawed data; moves to prevent access to climate data; and to keep research from climate sceptics out of the scientific literature.

As well as including new information about the emails, we allowed web users to annotate the manuscript to help us in our aim of creating the definitive account of the controversy. This was an attempt at a collaborative route to getting at the truth.

We hoped to approach that complete account by harnessing the expertise of people with a special knowledge of, or information about, the emails. We wanted the protagonists on all sides of the debate to be involved, as well as people with expertise about the events and the science being described or more generally about the ethics of science. The only conditions are the comments abide by our community guidelines and add to the total knowledge or understanding of the events.

The annotations - and the real name of the commenter - were added to the manuscript, initially in private. The most insightful comments were then added to a public version of the manuscript. We hoped the process will be a form of peer review.

This story is dark; there are no heroes. Environmentalists will be distressed at what happens in the labs; many may think we should not publish for fear of wrecking the already battered cause of fighting climate change. But some of it, according to the British government's Information Commissioner, may have been illegal.

Remember two other things. First, this was war. The scientists were under intense and prolonged attack, they believed, from politically and commercially motivated people who wanted to prevent them from doing their science and trash their work. And they had, as their most vocal protagonist Professor Michael Mann puts it in one email, "dirty laundry one doesn't want to fall into the hands of those who might potentially try to distort things ..."

Meanwhile, their attackers came to believe that the scientists were fraudsters. In many ways, what follows is a Shakespearean tragedy of misunderstood motives.

There are two competing analyses of what "climategate" means. One sees it as the mob entering the lab – the story of a malicious attempt to disrupt, cross-question, belittle and trash the work of mainstream scientists. This may or may not have been the motivation for the original hack, but it has certainly been the motive of some who have driven the news agenda since.

The second analysis sees it as democracy in action – the outcome of an entirely laudable effort by amateur scientists and others outside the scientific mainstream, headed by Canadian mathematician Steve McIntyre, to gain access to the complex data sets behind some of the climate scientists' conclusions, and to subject them to their own analysis.

The interweaving of these two narratives has created the tragedy of climategate. The bunker mentality of climate scientists such as the key email correspondents – headed by the director of the Climatic Research Unit (CRU) at the University of East Anglia, Phil Jones – is exposed in the emails. But so too is the chaos caused in the labs by the efforts of outsiders to question what was going on, without using the established rules of science, like working through publication in peer-reviewed literature. The clash of cultures between the blogosphere and the pages of august journals such as Nature could not be greater.

All this happened against the backdrop of a long-term assault by politically motivated, and commercially funded, climate-change deniers against the activities of many of the key scientists featuring in the emails. Indeed it is striking that people with a limited scientific involvement with CRU who have been victims of past attacks – such as Kevin Trenberth of the US government's National Centre for Atmospheric Research (NCAR) and Ben Santer of the Lawrence Livermore National Laboratory – became regular email correspondents with Jones and his colleagues. They were huddling together in the storm.

Through the emails we also see that some insiders were always demanding more openness from their colleagues and providing candid criticism of shoddy or mistaken work. One person stands out in this: Tom Wigley. He was Jones's former boss, having preceded him as head of CRU. Now based at the University Corporation for Atmospheric Research, in Boulder, Colorado, Wigley kept up a vigil for honesty and integrity in emails over many years. If there is a hero in this sorry tale, perhaps it is Wigley.

The science discussed in the emails is mostly from one small area of climate research — the taking of raw temperature data from thermometers, satellites and proxy measures of historical temperatures such as tree rings and turning it into useable information on temperature trends. The result being iconic graphs like the famous "hockey stick", first published 12 years ago and one of climate science's most famous and controversial products. It shows a long period of natural stable temperatures followed by a sharp, exceptional warming in the late 20th century.

In this area of work, CRU has been crucial. Under Jones's management, it has assembled the most comprehensive thermometer data record in the world, much of it under contract to the US Department of Energy. It is also home to some leading tree-ring researchers like the deputy head of the CRU, Dr Keith Briffa. The acerbic correspondence of Jones and Briffa with Michael Mann of Penn State University, the chief creator of the hockey stick graph, is a central feature of the emails.

CRU's work is the prime (though not the only) basis for the claim that man-made global warming is happening now and is exceptional in history. But as it comes under assault, it is worth remembering that it does not directly touch on other key issues like the physics of climate change, forecasts of future climate change and so on. Even if all the work of CRU were revealed as entirely phoney, which is far from being true, it would not demonstrate climate change was a hoax, or even much alter predictions of future climate.

The emails reveal that Jones, Briffa, Mann and other emailers were the gatekeepers of the science on which they worked. These men (there are virtually no women in the emails) reviewed papers by colleagues and rivals. They held key writing positions with the Intergovernmental Panel on Climate Change (IPCC) in its assessments of the science of climate change. So if they are damaged, then so is the IPCC.

Their correspondence reveals that there is some basis to the charge, made in October 2009 by climate contrarian Ross McKitrick, an environmental economist at the University of Guelph in Canada, that that "the IPCC review process is nothing at all like what the public has been told. Conflicts of interest are endemic, critical evidence is systematically ignored and there are no effective checks and balances against bias or distortion." There are more than a thousand leaked files of emails to and from scientists and CRU. The emails are clearly a small subset of all the emails that would have been sent and received by CRU scientists since the first one in 1996. Nobody is yet clear why this set made it into the public domain, but they are overwhelming between CRU scientists and foreign compatriots. They include technical discussions about tree ring chronologies and data analysis, scheming about how to repel Freedom of Information (FoI) requests, and bitching about their enemies among the sceptics – the group the scientists referred to as "the contrarians".

Our analysis finds previously undisclosed evidence of slipshod use of data and apparent efforts to cover that up. It also finds persistent efforts to censor work by climatic sceptics regarded as hostile – especially those outside the scientific priesthood of peer review – or those able to generate headlines in media outlets thought unfriendly, like Fox News.

We would agree with Judy Curry of the Georgia Institute of Technology, a leading climate scientist who maintains contacts with both camps, who says: "There are two broad issues raised by these emails ... lack of transparency in climate data, and 'tribalism' in some segments of the climate research community."

McIntyre's war

Climategate would not have happened without one man: a Canadian squash-playing blogger and data obsessive in his 60s called Steve McIntyre. Hero or villain, his data wars with Mann, Jones, Briffa and Santer largely created the siege mentality among the scientists, set them on a path of opposition to freedom of information, and by drawing in scores of data liberationists inside and outside the science community, almost certainly inspired whoever stole and released the emails.

McIntyre, a trained mathematician, had a successful career heading small Canadian minerals companies, often using his statistical prowess to analyse mineral prospecting data and out-bet his rivals. In 2002, he took up a new hobby – investigating climate change science. It started with an email from his home in Toronto to Jones at CRU asking for some weather station data. Initially the exchanges, as revealed on McIntyre's website ClimateAudit, were civilised. But as the years passed, and his data demands grew greater, relations soured.

From the start, McIntyre deconstructed studies that claim to show evidence of large-scale warming of the planet and of the human fingerprint in that warming. He pioneered the use of freedom of information legislation in the US and UK to demand the raw data behind the studies. It was not normal practice for scientists to publish this full data, nor the computer programmes they devised to analyse it.

McIntyre clearly doubted the statistical techniques being employed by the climatologists, and felt that, as a trained mathematician, he could do better despite his ignorance of climate science. And, as he grew more suspicious, he suspected them of cherry-picking data. He wondered exactly how Mann turned dozens of studies on the past climate, including a series of tree rings studies managed by Briffa at CRU, into his neat hockey stick graph. And he questioned the reliability of the thermometer data used by Jones to produce his graphs of warming over the past 160 years.

He found that no independent researchers had seriously tried to replicate the findings – a cornerstone of scientific inquiry. "Nobody's ever checked this stuff with any sort of due diligence," he said recently. He says too much is taken on trust in the cosy, collegiate world of science.

The climate scientists came to regard him as a meddling, time-wasting and probably politically motivated wrecker, who rarely published his own papers and devoted his retirement to trashing theirs. So when he tried to access their raw data and computer programmes, they resisted. The emails reveal that the researchers shared tactics, encouraged each other and competed for the rudest invective against McIntyre. And they grew even angrier as other wannabe investigators joined the data hunt. Men such as Doug Keenan, a former financial trader on Wall Street and the City of London, and a retired electrical engineer from Northampton called David Holland.

Many have accused McIntyre, Keenan and others of being hired hands of corporations out to fight climate change legislation. The Guardian has found no evidence of that. Instead, they appear to be an unanticipated outpost of the rise of "grey power", retired numerate professionals with time on their hands, an obsessive streak in their heads and a cause to pursue. The story of the battles of McIntyre and his acolytes to access the raw data, and the protracted and generally failed attempts by the scientists to repel him, is the central story of the leaked emails from 2003 onwards.

At first McIntyre published regular peer-reviewed scientific papers, co-authoring a couple with Ross McKitrick. The mainstream climate scientists responded angrily to them. They often used their influence to exclude what they regarded as substandard papers from major journals. So McIntyre, McKitrick and other sceptical authors, like Patrick Michaels of the University of Virginia and the Cato Institute and later Keenan, increasingly used Climate Research and Energy and Environment – two peer-reviewed journals widely disliked by mainstream climate scientists.

Tensions were strained further when McIntyre published more of his deconstructions of published papers on his website, but without scientific peer review.

Strident though his website often is, McIntyre has usually avoided outright personal abuse. The abuse was usually only a link away on other sites, however. And few of McIntyre's targets distinguished him from more politically motivated foes. Santer, for instance, concluded in one email in 2008 that McIntyre "has no interest in rational scientific discourse. He deals in the currency of threats and intimidation." He believes McIntyre saw himself as the "self-appointed Joe McCarthy of climate science".

Last September, RealClimate, a website run by Mann and other climate scientists, summed up how mainstream scientists felt about this kind of scientific discourse. "The timeline for these mini-blogstorms is always similar. An unverified accusation of malfeasance is based on nothing, and it is instantly telegraphed across the denial-o-sphere while being embellished along the way to apply to anything hockey-stick shaped and any and all scientists. The usual suspects become hysterical with glee that finally the 'hoax' has been revealed ... After a while it is clear that no scientific edifice has collapsed and the search goes on ... Net effect on lay people? Confusion. Net effect on science. Zip."

McIntyre, they complained, kept his hands relatively clean. He never talked about a hoax being exposed, and rarely questioned the "edifice" of climate science. He just picked away, providing fodder for his more excitable and less fastidious fans. As the RealClimate post went on: "Science is made up of people challenging assumptions and other people's results ... What is objectionable is the conflation of technical criticism with unsupported, unjustified and unverified accusations of scientific mal-conduct." McIntyre rarely makes such charges personally but, they complained, he "continues to take absolutely no responsibility for the ridiculous fantasies and exaggerations that his supporters broadcast".

There was a clash of cultures, too, between the ways of Canadian mining prospectors and those of academia. As one academic put it to me: "I think McIntyre confuses the more aggressive and confrontational style of business he used as a geophysical consultant with the more even responses in scholarship exchanges." On the other hand, the CRU emails hardly suggest that the scientists are shrinking violets. When Australian climate sceptic John Daly died, Jones commented, "In an odd way this is cheering news."

In the final months before climategate, the battle was not a cultural one, or even really about climate change. It was about data pure and simple. McIntyre wanted the scientists' data. In one week in the summer of 2009, he showered CRU with 58 freedom of information requests. He often made it clear that he did not have any particular reason for requiring the data. He just wanted to liberate it. It was a battle to break down the walls of the ivory towers, to blow apart the cosy world of peer review. It was a battle for the heart and soul of science, and for its lifeblood: data.

Then came the stolen emails. Whether hacked from outside or leaked from inside, the emails lit a fuse, but the fuel of mistrust had been piling up for years. As a result, the bonfire has been spectacular.

Scientists in the firing line

Many of the researchers caught up in the "climategate" saga have spent years in the firing line of sceptics. And they have felt the heat.

Kevin Trenberth had suffered abuse for publicly linking global warming to the exceptional 2005 Atlantic hurricane season, which culminated in hurricane Katrina. He told me: "The attacks on me are clearly designed to get me fired or to resign."

Ben Santer of the Lawrence Livermore laboratory in California, and formerly of the Climatic Research Unit at the University of East Anglia, was attacked for his role in writing the 1995 IPCC report, which claimed to see the hand of man in climate change. He said: "There is a strategy to single out individuals, tarnish them and try to bring the whole of science into disrepute."

Prof Mike Mann of Pennsylvania State University, fresh from his battle over the hockey stick in 2001, said: "There is an orchestrated campaign against the IPCC."

Funding trails to some of the more prominent sceptics also emerged at that time. Steve McIntyre, who runs the influential sceptic blog Climate Audit was free of financial conflicts of interest, but it emerged that prominent sceptic Patrick Michaels received hundreds of thousands of dollars in "consultancy" fees from the Intermountain Rural Electric Association, a coal-burning electric company based in Colorado. A leaked letter from the company's general manager, Stanley Lewandowski, said: "We believe it is necessary to support the scientific community that is willing to stand up against the alarmists."

The funding of climate sceptics has a long and probably ongoing history. In 1998, I revealed in the Guardian leaked documents showing that the powerful American Petroleum Institute (API) was planning to recruit a team of "independent scientists" to do battle against climatologists on global warming. The aim was to bolster a campaign to prevent the US government ratifying the Kyoto protocol.

The API's eight-page Global Climate Science Communications Plan said it aimed to change the US political climate so that "those promoting the Kyoto treaty on the basis of extant science appear to be out of touch with reality".

The leaked document said: "If we can show that science does not support the Kyoto treaty … this puts the US in a stronger moral position and frees its negotiators from the need to make concessions as a defence against perceived selfish economic concerns."

Its first task was to "identify, recruit and train a team of five independent scientists to participate in media outreach". It is not clear if the plan went ahead, but the policy objective was achieved.

Read the full manuscript of the Guardian's major investigation into the climate science emails stolen from the University of East Anglia and send in your own annotations to create the definitive account of the controversy