In a unique experiment, The Guardian published online the full manuscript of its major investigation into the climate science emails stolen from the University of East Anglia, which revealed apparent attempts to cover up flawed data; moves to prevent access to climate data; and to keep research from climate sceptics out of the scientific literature.

As well as including new information about the emails, we allowed web users to annotate the manuscript to help us in our aim of creating the definitive account of the controversy. This was an attempt at a collaborative route to getting at the truth.

We hoped to approach that complete account by harnessing the expertise of people with a special knowledge of, or information about, the emails. We wanted the protagonists on all sides of the debate to be involved, as well as people with expertise about the events and the science being described or more generally about the ethics of science. The only conditions are the comments abide by our community guidelines and add to the total knowledge or understanding of the events.

The annotations - and the real name of the commenter - were added to the manuscript, initially in private. The most insightful comments were then added to a public version of the manuscript. We hoped the process will be a form of peer review.

One man who has battled against climate sceptics longer than most is the climate modeller Ben Santer, who completed his PhD in climate science at the Climate Research Unit at the University of East Anglia in the 1983 before going to the Lawrence Livermore National Laboratory in California. He helped write the second IPCC assessment report, published in 1995. Most famously, this report claimed to find for the first time that "the balance of evidence suggests a discernible human influence on global climate" – essentially because the geographical pattern of warming across the globe matched that predicted by models. Santer was also lead author of some of the key research behind this claim and of the relevant IPCC chapter.

The main body of the report included lots of cavils about the claim to see the "discernible human influence". As another lead author on the critical chapter, Tim Barnett, then of the Scripps Institution of Oceanography, part of the University of California, San Diego, told me in 1996: "We wrote a long list of caveats in that chapter. We got a lot of static from within IPCC, from people who wanted to water down and delete some of those caveats. We had to work very hard to keep them all in." But many did not make it to the summary for policy-makers, and the New York Times leaked the text with the headline: "Scientists finally confirm human role in global warming." Nothing about the balance of evidence there.

Santer's co-authors on the critical research included Phil Jones at the Climatic Research Unit of the University of East Anglia, Tom Wigley at the National Centre for Atmospheric Research in Boulder, Colorado, and others. But as the main author of the research highlighted in the summary, Santer found himself under instant attack. Remembering all this a decade later, he told me: "I was accused of politically motivated tampering with the IPCC assessment, and of irregularities in my own research. I had, they said, somehow forced Nature to publish [my research] under duress, and had falsified scientific documents. There were calls for my dismissal."

His attackers were heavy hitters. Foremost among them was Pat Michaels of the University of Virginia, who kept a second office off-campus where he conducted his lobbying and public relations activities under the name of New Hope Environmental Services, an "advocacy science consulting firm". He has never disclosed who his clients are.

Michaels claimed that Santer had manipulated data in his critical paper. In particular, that he had ended his analysis of global warming patterns in 1987, just before a long surge in warming in the southern hemisphere, relative to the northern hemisphere, went into reverse. He claimed that this was contrary to model forecasts. He said that if Santer had included data from the next couple of years, which were available, it would have undermined the "discernible human influence".

Santer told me later that "Michaels had a legitimate scientific concern about the sensitivity of our results to the choice of data period". But he denied any "sinister purpose" and said that when he redid the analysis using the later data it "strengthened the original conclusions".

Others weighed in. Arthur B Robinson, a biochemist from Oregon, claimed that in the controversial paper, Santer and his co-authors had "deliberately omitted data points to create the trend that they reported... So Santer clearly faked the result, circulated it during IPCC proceedings in order to influence world global climate policy. They should never be permitted to work in science again." Robinson is an odd-ball. He is also a sceptic about Darwinian natural selection and has written a book about how to survive a nuclear war.

Wearing his other hat as IPCC author, Santer was also widely accused of being the man who added the key words "discernible human influence" to the body of the IPCC report, and of doing it very late in the day. [Ben Santer disputes this point and some others. See footnote] True enough. This was messy and does not reflect well on the IPCC. Those words were agreed at a main session of the IPCC in late 1995, attended by politicians. They wanted them included in the report's summary for policy-makers. But they went beyond what was said in the chapter from which the summary was supposedly drawn.

Yet IPCC procedure required that the chapters had to be made consistent with the summary, rather than vice versa. This is because the ultimate authors of the "intergovernmental" reports are the governments that approve the summary for policy makers. But such a rule puts the scientists in a difficult position, and Santer had the unenviable job of rewording his chapter to reflect the wording of the political summary. And of ensuring that all the authors were in agreement.

Santer told me the words were added to his chapter late, and without full consultation. But he said it was "essentially the same conclusion we [the authors of the chapter] had reached months earlier".

Critics point to a section of an earlier draft of the chapter that was deleted by Santer at this stage. It asked: "When will the detection and unambiguous attribution of human-induced climate change occur?" and answered "We do not know." But the contradiction is more apparent than real. Showing an "unambiguous" human impact is a much harder task than assessing the "balance of evidence". It was the assertion of a "balance of evidence" that Santer added.

The affair sounds like a semantic storm in a teacup. But it was exploited by political outsiders manoeuvring against the IPCC. Writing in the Wall Street Journal, Frederick Seitz, a physicist who headed the US National Academy of Sciences backed in the 1960s and later chaired the right-wing George C Marshall Institute, accused Santer of "the most disturbing corruption of the peer-review process" in 60 years.

The most unpleasant – and certainly for Santer most disturbing – language came from the Global Climate Coalition, a body representing the interests of the American oil and automobile industries. It accused Santer of "scientific cleansing" — a reference to the ethnic cleansing then going on in the Balkans. And for Santer, a Jew, it had another connotation. He told me in 2000: "My grandparents were subjected to ethnic cleansing. They died in a concentration camp in the second world war."

Santer spent months attempting to defend his reputation. He said later: "Nothing in your training prepares you for it. We are prepared for explaining our science, defending our science, and having scientists try to take your arguments apart. But we are not prepared for having our motives questioned and being accused of falsifying data. I think it is unproductive to engage with them directly. For many of them it is religious in a way. They are not rational. Don't waste our time; they don't have the same value system." This experience has coloured Santer's world ever since. It contributed to the break up of his marriage.

And in the leaked emails, he is seen sharing those experiences with other victims of hectoring and abuse by the more rabid climate sceptics. Others had their own horror stories, including Mike Mann over his hockey stick graph, Kevin Trenberth over his analysis of hurricanes and warming in the aftermath of Katrina, and later Jones over his escalating data wars. In each case, they argue, legitimate debates about scientific analysis and access to researchers' data have been turned into vindictive character assassination. A recurring theme of the CRU emails is how the researchers sought to avoid falling victims again.

Santer fights freedom of information request

In November 2008, Santer believed he was being dragged back into the front line, when he received an freedom of information request from sceptic Stephen McIntyre. At least one senior colleague and co-author on the paper in question thought Santer would be best advised to hand over the data. But Santer decided to fight the request.

It reconciled an apparent contradiction between surface temperatures as measured by Jones's thermometer network and satellite estimates of temperatures in the troposphere. While surface thermometers showed consistent warming, satellite and weather-balloon data suggested the warming did not extent up into the atmosphere. This was unexpected, since climate models suggested the opposite should be the case, especially in the tropics. It threatened to undermine Santer's "discernible human influence".

There were 17 authors involved in the paper in all, including Jones and Wigley. And the results mattered because a report for the US government published in April 2006 had highlighted the contradictory data as a "potentially serious inconsistency" in the science of climate change. The authors of had included many of the authors of the new paper, but also some of their arch foes.

But while Santer's team were assembling their paper, Santer received a copy of a rival paper from the journal Geophysical Research Letters, written by from David Douglass, an astrophysicist at the University of Rochester, New York, John Christy of the University of Alabama in Huntsville, and others. It highlighted the contrast between model findings and observational data in a way that suggested the models were wrong. Santer says he saw "serious scientific flaws" in the paper and recommended that the journal reject it. Which it did in September 2006.

Douglass persisted and produced a new version of his findings, published online at the International Journal of Climatology just over a year later in December 2007. It was widely publicised. Fox News reported it. Douglass told the National Press Club in Washington DC that it was "an inconvenient truth" about climate change, which proved that "nature rules the climate. Human-produced greenhouse gases are not responsible for global warming." The right-wing Heartland Institute took up the argument.

Santer regarded the paper as statistically flawed. Jones agreed. In an April 2007 email he wrote, "I know editors have difficulty finding reviewers, but letting this one pass is awful," said Jones. They went to war. Santer buttonholed Jones's colleague at CRU, Tim Osborn, a member of the editorial board of the journal. The authors were guilty of "intellectual dishonesty", he claimed in an email in January 2008. But he said a "quick publication of a response... would go some way to setting the record straight. I am troubled, however, by the very real possibility that Douglass et al will have the last word." To avoid that, he suggested that "our paper should be regarded as an independent contribution." They decided to redo much of Douglass's analysis.

Osborn contacted the journal editor, Glenn McGregor, a climatologist at the University of Auckland. Osborn later told Santer "he may be able to hold back the [print version] of Douglass et al, possibly so that any accepted Santer et al comment could appear alongside it." Osborn noted to Santer of their discussion the next day: "The only thing I didn't want to make more generally known was the suggestion that print publication of Douglass et al. might be delayed."

McGregor was probably somewhat nonplussed by all this. One of the people copied into the conspirators' emails had reviewed the Douglass paper for him, and had failed to raise any objection. Nonetheless, he agreed to a plan in which Santer et al produce their response as a paper, while the print version of Douglass et al was held back.

Santer's paper was published online, with 16 co-authors, in October 2008. And the two papers appeared together in the same print edition the following month. So, though both papers took about four months from submission to publication online, Douglass's paper took 11 months to get from online to print publication, while Santer's paper managed it in 36 days.

Nobody told Douglass and his colleagues about any of this. When the emails were published in November 2009, Douglass and Christy reacted angrily. They complained in the American Thinker in December 2009 about a surreptitious strategy involving the authors of the paper and the editors of the journal of "delaying [our paper] and not allowing [us] to have a simultaneous response to Santer et al."

At one level this is a matter of publishing etiquette. When is a response a paper? And what rules should govern responses to papers? But at another it is about power over the crucial scientific journals and the wider media.

There is no doubt the Santer and his colleagues sought to use the power they held to the utmost, albeit in a cause they regarded as in defence of good science. On the other hand, whatever the attempts to stage-manage publication, it was nothing compared to the stage-management of Douglass's paper in the media. It gained far more, and far more prominent, coverage than Santer's paper. In the world of science, Santer's team had the last word. Their charge that the statistical analysis in Douglass's paper was badly flawed and led to incorrect conclusions has, so far as the Guardian can establish, not been refuted. But Douglass got the publicity.

Or that is where the story stands now. For the affair lives on. With the Santer paper published, McIntyre weighed in. He asked for data from the 49 computer model runs conducted for the paper. Santer turned down McIntyre's request in an email on 10 November 2008. McIntyre responded with formal requests to Tom Karl at the National Climate Data Centre, where he guessed the data would have been held, and to the journal, saying Santer's response had been "discourteous".

The subsequent emails show Santer's rising concern that he faced a return to the nightmare of 1996. On 11 November he told Karl, who was one of the 16 authors: "I believe McIntyre is pursuing a calculated strategy to divert my attention and focus away from research... I believe that our community should no longer tolerate the behaviour of Mr McIntyre and his cronies." He called McIntyre the "self-appointed Joe McCarthy of climate science", adding: "We should be able to conduct our scientific research without constant fear of an 'audit' by Steven McIntyre; without having to weigh every word we write in every email we send to our scientific colleagues."

Many would see this as somewhat over the top, but given Santer's past experiences, it is at least understandable. And of course the reference to weighing every word in emails was rather prescient.

Santer wrote in an email on 3 December 2008 to Tom Wigley: "I'm damned and publicly vilified because I refused to provide McIntyre with the data he requested.... Had I acceded, I am convinced I would have spent years of my scientific career dealing with demands for further explanations, additional data, Fortran codes [a programming language] etc... For the remainder of my scientific career I'd like to dictate my own research agenda."

But after a further two weeks he had changed his mind, notifying the co-authors that he had decided to published online much of the data requested by McIntyre. He now reasoned: "This will make it difficult for McIntyre to continue making the bogus claim that he is being denied access to the climate model data necessary to evaluate the validity of our findings." Essentially he concluded that this was the path of least resistance, telling colleagues in January 2009 that "I agreed to this publication process primarily because I want to spend the next few years of my career doing research. I have no desire to be 'taken out' as scientist, and to be involved in years of litigation."

His change of mind brought a resounding slap on the back from Wigley, who had been working behind the scenes to persuade Santer, Jones and others to start releasing data, arguing that a spirit of openness would be beneficial all round. "Dear Ben," Wigley wrote a week before Christmas 2008. "This is a good idea. However will you give only tropical...results? I urge you to give data for other zones as well...To have these numbers on line would be a great benefit to the community. In other words, although prompted by McIntyre's request, you will actually be giving something to everyone."

He went on to ask "what period will you cover? Although for our paper we only give data from 1979 onwards, to give data for the full 20th century runs would be of great benefit to all... This is a lot of work — but the benefits to the community should be truly immense."

Keeping the public in the dark

Sometimes the scientists are exposed apparently trying to suppress inconvenient data from public attention in more popular presentations of their work. In 2008, Mick Kelly, a visiting fellow of CRU who is now based in New Zealand, discussed how to present that lack of recent warming to the public. In an email to Jones he discussed how he had "just updated my global temperature trend graphic for a public talk and noted that the level has really been quite stable since 2000 or so and 2008 doesn't look too hot." He said he anticipated that the sceptics will latch onto this quite soon" and suggested: "Maybe I'll cut the last few points off the filtered curve before I give the talk again."

Asked about this in December Kelly said "I didn't, of course, cut the points out... It was a joke, for God's sake. In future, I'll insert a smiley face to flag up humour."

Another email from the environment group WWF's Adam Markham in 1999 discussed fact sheets on climate impact risks in different countries being written by CRU for the environmental organisation. Markham suggested that the data on Australia was "slightly more conservative" than that coming from local scientists, and asked that it be "beefed up if possible". There is no record in the emails of whether Jones obliged, and Markham told the Guardian he cannot remember.

• This footnote was added 25 February 2010. Ben Santer disputes several aspects of the account given above. For his account of events and responses to some of the points in Fred Pearce's article, see this blog on RealClimate.org.

Read the full manuscript of the Guardian's major investigation into the climate science emails stolen from the University of East Anglia and send in your own annotations to create the definitive account of the controversy