So far, most of the Climategate attention has been on the emails in the data dump of November 19 (see here, here, and here), but the emails are only about 5 percent of the total. What does examining the other 95 percent tell us?

Here’s the short answer: it tells us that something went very wrong in the data management at the Climatic Research Unit.

We start with a file called “HARRY_READ_ME.txt.” This is a file containing notes of someone’s three-year effort to try to turn a pile of existing code and data into something useful. Who is Harry, you ask? Clearly, a skilled programmer with some expertise in data reduction, statistics, and climate science. Beyond that I won’t go. I’ve seen sites attributing this file to an identifiable person, but I don’t have any corroboration, and frankly the person who wrote these years of notes has suffered enough.

The story the file tells is of a programmer who started off with a collection of code and data — and the need to be able to replicate some results. The first entry:

(yes, they all have different name formats, and yes, one does begin ‘_’!)

Believe it or not, this tells us quite a bit. “Harry” is starting off with two large collections of data on a UNIX or UNIX-like system (forward slashes, the word “filesystem”) and only knows very generally what the data might be. He has copied it from where it was to a new location and started to work on it. Almost immediately, he notices a problem:

6. Temporarily abandoned 5., getting closer but there’s always another problem to be evaded. Instead, will try using rawtogrim.f90 to convert straight to GRIM. This will include non-land cells but for comparison purposes that shouldn’t be a big problem … [edit] noo, that’s not gonna work either, it asks for a “template grim filepath,” no idea what it wants (as usual) and a serach for files with “grim” or “template” in them does not bear useful fruit. As per usual. Giving up on this approach altogether.

Things aren’t going well. Harry is trying to reconstruct results that someone else obtained, using their files but without their help.

8. Had a hunt and found an identically-named temperature database file which did include normals lines at the start of every station. How handy — naming two different files with exactly the same name and relying on their location to differentiate! Aaarrgghh!! Re-ran anomdtb:

Okay, this isn’t so unusual, actually, but unless you document and describe your file structure, it’s pretty much opaque to a new reader. Still, Harry presses on:

11. Decided to concentrate on Norwich. Tim M uses Norwich as the example on the website, so we know it’s at (363,286). Wrote a prog to extract the relevant 1961-1970 series from the published output, the generated .glo files, and the published climatology. Prog is norwichtest.for. Prog also creates anomalies from the published data, and raw data from the generated .glo data. Then Matlab prog plotnorwich.m plots the data to allow comparisons. First result: works perfectly, except that the .glo data is all zeros. This means I still don’t understand the structure of the .glo files. Argh!

Thanx for this. Interesting. However, I do not think your
response is very good. Further, there are grammatical and
text errors, and (shocking!!) you have spelled McKitrick
wrong. This is a sure way to piss them off.

They claim that three cores do not cross-date for TRW.
They also say (without results) that the same applies to MXD
(these results may be in their Supp. Mat. — I presume you
checked this).

So, all you need say is …

(1) TRW was not the only data used for cross-dating.
(2) When MXD is used there are clear t-value peaks,
contrary to their claim. You can show your Fig. 4 to prove
this.
(3) The 3-core-composite cross-dates with other (well-dated)
chronologies (Yamal and Polurula), confirming the MXD-based
dating. You can show your Fig. 5 to prove this.

You could say all this in very few words — not many more than
I have used above. As it is, your verbosity will leave any reader
lost.

THERE ARE SOME PROBLEMS STILL. I NOTE THAT 1032 IS NOT COLD IN YAMAL.
SEEMS ODD. IS IT COLD IN *ALL* OF THE THREE CHRONOLOGIES AT ISSUE?
OR DID A REINDEER CRAP NEXT TO ONE OF THE TREES?

Also, there seems to be a one-year offset in the 1020s in your
Fig. 6.

I hope this is useful. I really think you have to do (and can do) a
better job in combatting the two Ms. If this stuff gets into Nature,
you still have a chance to improve it. Personally, I think it would
be good for it to appear since, with an improved response, you can
make MM look like ignorant idiots.

This is not, sadly, all that unusual. Simply put, scientists aren’t software engineers.

When one’s institute is maintaining the database and code used to inform political debate as well as be the basis for many, many other researchers this excuse just doesn’t cut it.

We have to use ISO 9003 standards for FDA work and I would argue that this work is every bit as important as a fragment of image processing software running on some CT workstation.

Every paper from CRU and every paper that depends upon CRU datasets/models is now invalid until the research can be replicated. This includes the IPCC reports btw. Furthermore, any paper that cites these papers should be reviewed by its authors to see if their conclusions were invalidated because of dependence upon this scientific misconduct.

NASA may well be as screwed up as CRU. They have resisted for years releasing their data and code. In the face of this corruption of science, there can be no reason that ANY publicly funded dataset or model is kept secret.

Furthermore, the datasets and models used for environmental science in particular should be put into the open source community for review and comment. If this piece of junk that that poor b@sta@rd had to deal with had been available to thousands of skilled engineers and scientists he might have not wasted 3 years of his life attempting to resurrect this travesty.

So, with millions of dollars of taxpayer money in their pockets – traveling to exotic locales for seminars and conferences – eating steak and lobster- and with trillions of dollars on the line globally and with the economy of every nation on earth on the line… they can’t make their own data work no matter how they go about it???

Why do those people still have jobs? Why are they still getting my money?

This sort of mess is apparently fairly common in graduate departments (I assume CRU is such), as students do what is necessary for their work, then move on.

That doesn’t excuse CRU, however, because of the use that is made of the data. The fact that nobody else has the data, and CRU doesn’t have it in a usable form (and may never be able to decipher it) means that, for scientific purposes, the data doesn’t exist.

This is not, sadly, all that unusual. Simply put, scientists aren’t software engineers.

Nor are they engineering project managers. A PM would have realized that this was a show stopper, and halted everything until this could have been straightened out.

But you don’t have to read far into the emails to realize that Jones frankly didn’t care whether his science was right. Even if this programming fubar hadn’t have happened due to personnel issues, he (and his American counterpart Mann) simply was a man of low ethics. When you have low ethical standards, sooner or later things spin out of control.

To paraphrase a post I made elsewhere, the code shows that their “models” are not models at all in the sense that physical scientists use that
word, i.e., simulations of physical processes. They are extrapolations of historical data taken from a variety of (in many cases) diverse sources that are then statistically manipulated and tweaked. That is why the programmer is having so much trouble, i.e., there are no physical laws
to refer back to, e.g., there is no way there could be an “energy budget balance” problem if they were doing physical modeling.

AGW and the Global Cooling,oops I mean,Warming, oops I mean,Cooling, etc. has the same problem most politically motivated people have. If you want your report or survey or study to come up with an answer before you even start, you sometimes have to just make sh*t up.

People have an effect on the planet, no doubt. Let’s don’t pollute if can can avoid it. Let’s find newer cleaner sources of energy….but LET’S NOT place our faith and futures in the hands of socialized and politicized scientists that have been less than honest about their theories than they would have us believe.

I hope these emails and data knock the AGW people off the rails. But, I hope we, as a group, continue to try and be as careful as we can with our world.

I have the impression that their statistical toolkit is rather limited, mostly PCR and PCA, i.e., fancy dress cousins of least squares polynomial fits. Then they throw out lots of ‘outliers’ and such. This makes extrapolation rather silly, because what they have isn’t a model in any normal sense of the word. And making an error estimate for the result can be rather tricky because of the ad hoc decisions about which outliers to eliminate. The divergence “problem” isn’t really a problem, it’s merely an indication that the idea didn’t pan out and it’s time to find another thesis topic. But it’s a problem for these guys because they can’t think of anything else to do.

That’s my impression, anyway. I am not, and never have been, a climate scientist.

The CRU emails show the activities of the small clique of researchers at the heart of AGW research. Here is an excerpt from Wegman’s 2006 social network analysis of temperature reconstruction, annotated with links to the searchable archive of CRU emails:

One of the interesting questions associated with the ‘hockey stick controversy’ are the relationships among the authors and consequently how confident one can be in the peer review process. In particular, if there is a tight relationship among the authors and there are not a large number of individuals engaged in a particular topic area, then one may suspect that the peer review process does not fully vet papers before they are published. Indeed, a common practice among associate editors for scholarly journals is to look in the list of references for a submitted paper to see who else is writing in a given area and thus who might legitimately be called on to provide knowledgeable peer review. Of course, if a given discipline area is small and the authors in the area are tightly coupled, then this process is likely to turn up very sympathetic referees. These referees may have coauthored other papers with a given author. They may believe they know that author’s other writings well enough that errors can continue to propagate and indeed be reinforced….

As a former analyst and programmer, if extracted data used does not produce the desired results, as in accounting, a 999,999,999 card can be used to balance a worksheet and account for missing or incomplete input.

The extent of the lies and deceit exposed by the AGW believers is astounding but not unexpected. We are on the brink of another major financial disaster and if our leaders in Washington have any sense of decency, this madness will stop now. However, after witnessing the psychosis permeating our elected representatives once they become entrenched, hope for sanity and common sense is fleeting.

We have a cluster of barn owls who truly believe they possess the intelligence to guide us into the future when all along, all they produce is owl crap and take flight when confronted with truth.

There is absolutely no reason for anyone in a position of authority to have a moment’s regret or delay in PRYING these money-sucking leeches out of whatever lab they’ve stashed themselves in, take their keys and credit cards, give them $100 pocket money and a $50 gift card for Walmart, change the locks and tell them to get lost. Permanently.

11. Desert Dog:
People have an effect on the planet, no doubt. Let’s don’t pollute if can can avoid it. Let’s find newer cleaner sources of energy….but LET’S NOT place our faith and futures in the hands of socialized and politicized scientists that have been less than honest about their theories than they would have us believe.
___________________________________________________

Sir you are exactly right!
I just wrote an article about this on Searchwarp…a writers blog:

This whole thing provides me with scads of entertainment. If you don’t have the entire set of documents, get them. You don’t need to be a climate scientist to know the smell of poo. AGW jumped the shark a long time ago.

If these folks had any *real* intelligence, they would not record anything against self-interest in e-mails, they would not have open discussions about their lousy programming skills, and they would not have been so d@mn cocksure that such arrogant stridency wouldn’t be the shoelace that leads them to trip up (or rather have a 3rd party tie their laces tied together).

They became zealots in a religion where they were the prophets, where they were assured of a decent living through grant money and speaking engagements, and where their questioners were ruthlessly skewered to their great amusement.

Make ‘em walk the plank!

Also, I find it mildly amusing that the New York Times, publisher of the Pentagon Papers, and about two dozen other classified documents over the past three-plus decades, suddenly stands tall as the protector of the Climate-gate e-mails. To paraphrase, the e-mails were not intended for public eyes.

Apropos #13, it well known among authors of scientific papers that many editors choose the reviewers from among the references. Thus, it is possible for the authors to, in some sense, pre-select the possible reviewers by careful tailoring of the references. This coupled with the lack of blind reviewing seriously calls into question the scientific peer review process in many cases.

amazing. as big a story as this is, the media puts their head in the sand and doesn’t report it. secondly, sites like Charles Johnson Little Green Footballs treat it as a non-story-since they believe the global warming narrative. LGF goes so far as to defend the hypocrisy of the scientists!

“They clearly have some history of massaging the data — hell, practically water-boarding the data — to get it to fit their other results.”

Bravo! And I think you’re exactly right about the data-corruption being the most damaging aspect of the ClimateGate scandal.

The entire “climate modeling” racket was built on corrupted data, corrupted programs, and corrupted researchers. And upon climate modeling stands the whole Climate Catastrophe Industry: a multi-billion-dollar, pseudo-scientific house of cards.

The previous revelations plus reading through “Harry’s” emails gives one the eerie feeling of being a front-bench witness to the greatest slow-motion train wreck in history.

Initially I was cautious. The original email files looked damning on the surface but as many noted, some of the phraseology could be taken several ways depending on whether one was supportive or pejorative. But the analysis now coming out of the data sets and program files seems to confirm the worst. Now the data/program files confirm the real intent of the emails and the emails confirm the fraudulent intent of the data manipulation; they cross-confirm each other.

Scientists assemble data, do their analysis, get the results and then go where the results take them. These crooks assembled the data, did the analysis, and then hacked it all to make the results go where they wanted them to. –Every last jot and titter of it by the way it’s beginning to look.

Wow. Just WOW !

15 Bart: Sorry buddy. Our leaders in Washington have no sense of decency. Stronger measures will be required.

I went to check the growth rings in those crucial Siberian trees, but the trees were gone. Some high flautin Englishman had cut them into fire wood and set them alight, saying the atmosphere needed some warming.

Bugs or no bugs, why does it matter? As long as the “scientists” could scam multimillions, and billions. A functional pgm would never produce the desired results. A programmer who wrote a correct program would have lost his job a long time ago.

Bush politicized science to deny the reality of man made global warming.

Al Gore, the soon to be billionaire, makes his billions on global warming. Should he give his Nobel back, or donate the prize money to charity, such as ACORN?

Neko-san, when I worked as an editor for reviewed journal, I looked for honest people outside the discipline for exactly that reason: bias. Didn’t want it, otherwise what was the use of publishing? Then I attended a technical journal conference and learned that the built-in biases and blinders do intentionally exist in some disciplines. Sad but true.

This is not, sadly, all that unusual. Simply put, scientists aren’t software engineers.

Yeah, but they are scientists, or at least they’re supposed to be. And scientists have to understand the data they’re using. They may not understand programming very well, they may not know how to use VCS or buddy testing or unit tests or code reviews or inline comments or any other common practices for computer engineering. But they ought to know enough to document their data – HARRY_READ_ME.txt shows that not only did nobody know how the software worked, they’d lost the metadata about a bunch of their data files. Poor Harry was reduced to guessing which columns might be longitude, and not able to find latitude at all.

(hmm, I’m no climatologist, but I have a sneaking suspicion that latitude might be kind of significant to temperature data sets. But then I haven’t had any papers published in the clique- peer-reviewed press, so what do I know)

My advice to the British Crown would be to shut down this hack of an institution, close the doors to CRU, fire the entire staff, delete all of the data, demolish the buildings, and forget it ever existed. Hire some real scientists to start a real research center to do real research with real data. If they start now, in ten years they might have something useful to report.

Over at the BBC Science and Environment webpage, Roger Harrabin, the BBC’s Environment Analyst (whatever the heck that is) has had to address the CRU hacked documents. Harrabin does not allow readers to comment, and in the story he provides ample opportunity for the CRU proponents of AGW to state their case, and to essentialy claim this is much ado about nothing. Specifically, allow me to quote this bit which Harrabin tells us comes from an inside source of his at the CRU:

“My CRU source points out that its unpublished full data set is almost identical to the ones at the National Climatic Data Center and the Goddard Institute of Space Studies.
Both of these are in the US, where there are no restrictions on publication. The CRU view is that when the sceptics see the full data in due course they will be very disappointed.”

The BBC’s Harrabin concludes by mentioning that about AGW, “One leading figure told me unofficially that confidence was now at 99%.”

I think that’s the data you and “Harry” are agonizingly trying to analyze in this post Charlie.

Linking to Chris Horner writing for the American Spectator we get this from Clarice’s post:

“Today, on behalf of the Competitive Enterprise Institute, I filed three Notices of Intent to File Suit against NASA and its Goddard Institute for Space Studies (GISS), for those bodies’ refusal – for nearly three years – to provide documents requested under the Freedom of Information Act.”

Interesting.

On that side of the pond CRU is saying the data of the Goddard Institute of Space Studies, (GISS) is aparantly open and transparant for publication, while on this side of the pond that very same (GISS) is being sued for 3 years worth of hiding it’s data from FOI requests.

Now I do not know specifically what information is being demanded of NASA and the GISS in this Lawsuit, but wouldn’t it be interesting if it’s essentially the exact same data hacked from CRU.

If so, then the supposedly open to publication sources on our side of the pond are precisely what the CRU are using in Britain to attempt to quiet the firestorm and justify the legitimacy of their research over there. Would love it if it’s exactly the stuff that Clarice’s column here addresses as being the data that has been denied review by independent researchers in America.

If so, this would appear to me to be science scholarship done on the level of Ward Churchill.

Simply post bogus research, (hidden from outsiders) on both sides of the pond, then refer to the publication of that bogus research across the pond as validation for your own bogus research. Keep it all secret from independent researchers worldwide, and then have the BBC Environmental lapdog Reporters, who disallow comments BTW, pronounce that their ‘in the know, top level sources’ swear that “it’s 99% absolutely true,” so On to Copenhagen!

Calvin Ball wrote: “When you have low ethical standards, sooner or later things spin out of control.” Ten for ten!! Or, as his baby sitter said, ” Calvin, I think you’re just making this up a you go along.” “Hobbes! She’s stumbled into the perimeter of wisdom! Run!!”

My advice to the British Crown would be to shut down this hack of an institution, close the doors to CRU, fire the entire staff, delete all of the data, demolish the buildings, and forget it ever existed.

I agree except for the following minimal changes

My advice to the British Crown would be to shut down this hack of an institution, close the doors to CRU, fire the entire staff, shoot the entire staff as a warning to traitors, delete all of the data, demolish the buildings demolish the buildings and seed the gound with salt, and forget it ever existed.

The quality of their data and code might make more sense if one considers it in the context of a business instead of science. Need to get the product out, there are TTM pressures you know. Get that revenue cycle started, and we’ll fix it in production!

As an engineer it was obvious to me from the start that a system as complex as the climate of a planet was not something that could be modeled with any accuracy. Our experience is too limited, our time scale too short and knowledge too feeble. Indeed, one of the first things you are taught in engineering schools is that you cannot directly solve for temperatures in open or highly complex systems. Belief in AGW is almost non-existent amongst engineers for this very reason.

What I did not realize is that no attempt at modeling had even been attempted. Apparently all they did were statistical projections of trends in observed temperature proxies. There were no physical laws included whatsoever. This is in and of itself enough to invalidate their claims of “modeling” before we even get to the fact that their data is worthless and they can’t even reproduce their own results.

What they did was the equivalent of putting a wind gauge next to a highway as a means of guessing how fast the cars were traveling. Then taking that proxy data and extrapolating from there in whatever implausible way was necessary to generate what they wanted to see. But alas for them, one day the wind stopped blowing and their little cars stopped moving… thus proving their approach was worthless from the start.

We are lead by sub-moronic, scientifically illiterate people such as Nancy (natural gas isn’t a fossil fuel) Pelosi and Al (the center of the earth is millions of degrees) Gore and we act like they have some right to use force to restructure every aspect of our economy and our existence just by virtue of the positions they hold. Such a thing cannot help but end badly if they are allowed to have their way. The Constitution was written to limit the power of exactly this these sorts of people.

I don’t think it is much of an exaggeration to say that whoever released this information may have saved more blood and treasure than anyone else who has yet lived in this century or perhaps even the last century. Monuments need to be built to this man or woman.

If it doesn’t fit, aquit –
And if the facts don’t fit your theory, adjust them to fit -
In OJ Simpson case, in preparing the final “glove test” trick (OJ suffering of arthritis), the defense team stopped him for weeks to take his medicine -
The result – swollen hand(s) which simply the glove coudn’t accomodate – aquit!

Let’s not overlook the even larger problem at CRU. CRU is the repository for much of the world-wide historical temperature database. There are statements in these documents that indicate that these databases have been corrupted, with much of the metadata that describes location of the measurements missing, as though the original archived data files were used for calculations instead of copies of the files. This might be a far more important problem caused by mismanagement at CRU that affects a far wider range of potential scientific investigation.

My doubts about man-made global warming have increased considerably in the last couple years and these hacked emails simply add to my skepticism. Now to identify the person or group who can speak clearly on the matter. 80% of the messages here slip into rant and bluster before the final period is placed.
Conservatives tend to take the approach that ANY regulation of the environment is bad because it gores the profits of their industrial pals and expands the role of government. Lefties go the other way, sometimes just to stick it to said industries.
In my view, said industries DO need a lot of regulation, but not based on tree rings from Siberia and unfathomable models, but rather clearly discernible effects on our land, water, and air. There will always be strong differences on the relative value of jobs vs spotted owls and snail-darters, and plenty of accompanying bluster, but I crave the clear-eyed centrist who can sort through this stuff.

Re #19 Mhu Cao: “If these folks had any *real* intelligence, they would not record anything against self-interest in e-mails, they would not have open discussions about their lousy programming skills, and they would not have been so d@mn cocksure that such arrogant stridency wouldn’t be the shoelace that leads them to trip up (or rather have a 3rd party tie their laces tied together).”

Good point that’s obviously lost on many. Considering the possibility that these guys aren’t complete morons, you wonder about all of the correspondence where they were smart enough not to put it in writing, or were smart enough to delete after the fact.

My advice to the British Crown would be to shut down this hack of an institution, close the doors to CRU, fire the entire staff, shoot the entire staff as a warning to traitors, delete all of the data, demolish the buildings demolish the buildings and seed the gound with salt, and forget it ever existed.

#46, It is not that making a physical model is impossible, but it is a very computationally intensive problem. For example, there are roughly
510M sq. km of surface area and the atmosphere is roughly 120 km high, so a model with 1km cubes (actually upside down truncated pyramids) would have more than 6G cells. And that is just the air!

Since heat transfer with the surface (oceans and land) would have to be also modeled, figure another 6G cells below the surface. Plus land and oceans need to be modeled differently, and the boundary between them is irregular. And, then, what time step does one use?

While that might sound within reach now, how much memory did the average computer have 10 years ago and how fast was the cpu? The problem would have been challenging even for supercomputers then.

To make it fit, one could increase the cell size. But as the cell size increases, that tends to obscure the potential effect of local features (like mountain ranges). The same with increasing the size of the time
step, e.g., on the scale of even one year, there are no hurricanes, tornados or monsoons.

Another way to simplify is to assume some sort of symmetry, but that
becomes increasingly unrealistic for simulating climate (though it works well for other kinds of problems).

The CRU scandal is the scientific version of the Madoff scandal. In both instances, billions of dollars are/were at risk. The political ramifications of the Madoff scandal favored rapid resolution and prosecution: Unfortunately, that is not the case where AGW is concerned. Furthermore, were see the ability of politics and its largesse to corrupt science.

To Dwight @ 57 Are you familiar with the file labeled “The Rules of the Game”; this looks to be a conference hand-out, pamphlet… It leads with the observation “Changing attitudes towards climate change is not like selling a particular brand of soap-it’s like convincing someone to use soap for the first time.” Is this how you feel?

Everyone has to realize, we know NOTHING about climate except by anecdote. What we thought we knew, now we can’t be sure.

The data is no good—much of it messy to begin with, and now hopelessly corrupted. Wherever there may be good, uncorrupted data, how can you tell the difference?

The models are no good. They can’t reproduce the past even with spoon-fed data and faked up parameters. One model even gives negative numbers for a sum-squares calculation.

We have to start over. Basic data in its original form, reviewed by honest scientists with forensic auditors by their side; and changes or adjustments discussed fully and transparently. All data subject to chain of custody just like evidence in a legal proceeding.

Models reviewed at the source code level, to ensure they are properly implementing good science. All parameters must be justified. Models, too, must be secured from tampering.

New programs to collect data going forward, including all handling of that data and any adjustments. Not just CO2 or GHGs, that limitation prejudges the result—must include EVERYTHING that could conceivably affect climate.

Eventually, bring the validated data into the validated models, and let’s see what we get and discuss where we have to go to improve the science and the modeling. And only at that point does any talk about public policy come back into play.

I guess 3-5 years if we start NOW. And anyone who objects we can’t waste that time, the threat is too grave and too imminent needs to be reminded that they have to unlearn what they have been told, because we actually now know almost nothing. Give ‘em some thorazine and send them to their rooms to grow up.

Meanwhile, Mann, Jones and most of this crew needs to spend their time defending themselves in court, before administrative panels, and academic discipline boards.

#58 Tom wrote: “Changing attitudes towards climate change is not like selling a particular brand of soap-it’s like convincing someone to use soap for the first time.” Is this how you feel?
————–
What? I don’t follow your context here.

As a software engineer, i am worried by the lack of rigour in the keeping of their software code. If they cannot process their data properly, let alone leave it undocumented and unmaintainable, how can the public accept that their results are valid??

HARRY_READ_ME.txt indicates very serious problems with that data set. If it were just one small study, one small study group, it would be one thing; but these CRU boys are big players.

Simply: Kyoto/Copenhagen and cap/trade are over-leveraged on a data set that appears to be crumbling. Danger ahead, for Democrat or Republican, mathematician or carpenter, doctor, lawyer or Indian chief.

As a trained engineer I can assure you that if you torture the data long enough…it will confess to anything. How can it be that despite the enormous significance of this scandal, so few in the MSM are covering it…at all.

Exactly. What the code comments tell us is that one of the major repositories for climate datasets in the world is hopelessly corrupted. Once those annotations are lost and the datasets become unknown then there is no reason to trust them any longer. And, given the fraudulent behavior of the warming advocates at CRU, no reason to believe the datasets haven’t been tampered with.

Hopefully the datasets can be recreated from raw data stored on tape/card/paper. If it is, the validation and verification process needs to be audited.

We still need to understand climate and climate research is the best way to do that. Next time maybe society can prevent this field of study from being suborned by a collection of unethical fanatics.

@45, for what it’s worth, there are plenty of business domains where you have to provide validated, controlled, audited data (pharmaceuticals or airplane repairs for example). A regulated business with this level of data quality would have been shut down long ago.

Let me repeat that. A business with a data shop like the CRU would not be allowed to sell cornflakes, much less reorganize the entire global economy.

In a different thread, I sarcastically suggested that the climate “models” are like programs to predict a roulette wheel based on data about the history of the wheel, i.e., the result of past spins.

In retrospect and without the sarcasm, I think a more correct analogy is a program to predict stock market prices. Stock prices follow a random walk on a small scale but also can follow longer term trends. Right up until the point they don’t because of the influence of unpredictable outside forces (like a sudden liquidity crisis among financial institutions).

Similarly climate follows predictable trends right up until the point it doesn’t, e.g., Roman Warm Period, Dark Ages Cold Period, Medieval Warm Period, Little Ice Age, etc. The interesting question is why the transitions from warmer to colder or colder to warmer occur, but this
depends on little understood physical processes that climate “models” don’t address.

The problem with climate “models” isn’t just that the data has been lost or corrupted, the results not reconstructable, the programs broken, the raw data “adjusted” to fit the conclusion, etc., though those are all certainly problems; the real problem is that there is no scientific basis for the methodology at all.

Let me give you an example from finance to show the contrast. In securities analysis, the CSRP at the University of Chicago is considered the definitive source of data and is available to all researchers for a nominal fee. It is generated from published market prices and easy to verify, although what makes CSRP so good is that it has data on stocks that have been delisted due to mergers, bankruptcies, etc. So it’s really raw data.

Then, any findings generated using the data are easy to duplicate and verify, as papers are expected to provide enough information to make this possible. Also finance research is historical and makes no claims about the future – you as an investor might assume a 10-year, 20-year, or 40-year trend will continue, but that’s up to you. The research certainly doesn’t make those claims.

In contrast, this bogus climate research seems to use a proprietary closed data source, a proprietary closed model, and does make claims about the future. And although I don’t know much about climate science, it seems like reading a historical trend of station temps could be done on a home computer by the average college student – its not rocket science.

Predicting future financial performance involves decision making under uncertainty and predicting climate (like the 7- or 10- day forecasts) is probably similar. The way that’s done is you run a number of scenarios, called Monte Carlo. Its imperfect by definition and you’re wrong more often than you’re right – its really about determining what scenarios are theoretically possible rather than what’s going to happen. Those predictive models are heavily influenced by inputs of course. To really evaluate your model, you need to be able to evaluate and confirm past predictions over hundreds of trials.

Then there is Greenland where the ice cover is rapidly melting. But the model doesn’t give the expected results!!. Damned but what the f*ck and meh, lets hammer the precipitation grid in shape around the southern parts of Greenland. And Presto we have a new “scientific” reality.

From Harry’s README line 1496:

—- Firstly, wrote mmeangrid.for and cmpmgrids.m to get a visual comparison of old and new precip grids (old being CRU TS 2.10). This showed variations in ‘expected’ areas where changes had been made, it the Southern tip of Greenland. —

Right now it seems the greater threat to mankind are these scientists. A time line of emails , data and code should be laid out and then an attempt should be made to find real input data and workable models that have statistical validity. Right now the input data sounds ridiculously fragile / tortured and thus I doubt the models and their predictive ability would be worth much in the light of day. What a weird group of so called scientists!

Here is my theory of what happened. Britain has a FOI law. Phil Jones was conspiring to violate the FOI law by deleting files and emails. Some poor IT shmuck was directed by Dr.Jones to do the deleting.

The poor IT shmuck was faced with the unenviable choice of deleting the files and email with the significant possibility that he would go to prison while Dr.Jones washed his hands and did nothing or the poor IT shmuck could refuse to delete the files and get fired into today’s very unfriendly job market.

The poor IT shmuck resolved his dilemma by leaking the emails, software and data which is going to get Dr.Jones fired instead.

If you are working in IT at some institution that is also faking climate data and is liable to FOI requests, you had better be maintaining your own zip file if you would like to stay out of prison.

If this was part of a Bones episode or something people would scratch their heads and say ah-ha…but in reality it’s been drummed into so many heads by the MSM it’s gonna take a’while to get through to them. More satire from Daily Show and SNL, as well as Jay Leno might help change their opinion…

As someone with a MS in meteorology and “training” in programming, I can tell you this is quite common in the meteorological community, esp. in academic and research circles. I have a lot of code that “works” but I wouldn’t bet multiple terabucks on the outputs. With all the money being flushed into climate research, it’s a scandal that software engineering is not demanded through the process.

Outside auditing of the entire process should be a requirement. “Peer review” is not an audit. If you want to make trillion-dollar changes in the economy, you should spend a few mil on proper software principles and be able to repeat your processes end-to-end. The fact that poor Harry couldn’t duplicate the results, or even understand how someone else got to the results he was handed, shows you can’t trust them.

‘We argue that the evolving practice of science in the contemporary world must be different from the classic view of disinterested – almost robotic – humans establishing objective claims to universal truth.’

I try to keep an open mind on these issues, and am accordingly receptive to new information and ideas. The issue here is honesty. One of the reasons I became a global cooling skeptic was that for years I would read the global warming deniers’ writings on the web, and then refer to the scientific data they were commenting on. And they didn’t match.

For example the Heartland Institute once published a broadside on the melting of the Arctic icecap, quoting a press release of the US oceanographic institute, NOAH I think it’s called. I went to the NOAH web site and discovered to my dismay that the Heartland Institute was lying through its teeth. Its claims were not in the slightest borne out by the NOAH report.

I can search my files and give you the exact dates and cites, if you don’t believe me.

In any case, I would recommend that you devote at least a portion of your skepticism to those in your own ranks who produce nothing but lying propaganda. I think that would make global warming denial a considerably more respectable pursuit than is is at present.

No matter what is discovered about the climate fraudsters, they will gleefully try to convince us that the darkness outside is really daylight…Really? Well this time fewer are buying it. .We “knowNothings Flatearthers” (REALISTS) have known for a long time that phony demagogues on the left will stop at nothing to lead the naive, the ill-informed, the un-informed and the mis-informed down this disastrous superhighway of deceit, whether it be single payer health care, cap/trade, or CAFE standards for cars. If they really cared about greenhouse gases, they would shut their traps and include nuclear power and NatGas on the menu. But they don’t so, we REALISTS must conclude it’s money and control they are after…With the latter being the main dish..HPD out….

The quality of their code & data might make more sense if one considers it in the context of a business rather than science. Need to get the product out, there are TTM pressures you know. Get that revenue cycle started, and fix it in production!

Harry is not a professional FORTRAN programmer. The forrtl: error (75): floating point exception usually division by zero or overflow. In this case, it is probably an overflow, hence the large negative number in OpTotSq. In the early days some computers had 16 or 32 bit registers only. The leftmost bit was reserved for sign bit to indicate positive or negative number. However, if a number was defined as an Integer and was too large , eg. for a 16 bit register larger than 65536, the leftmost bit of 0 (positive number) was changed to 1 ( negative number ) hence the large negative result.
This is FORTRAN 101. Harry must have been a scientist-turned-programmer without a proper training.
Of course, I would have to see the code to be 100%, but most probably that is what happened. The OpTotSq should have been defined as a floating point number and not as an Integer. That is why it worked before, the number was less than for example 65536 for 16 bit machine. Once over 65536, it became negative.