Another Assignment for the EAS8001 Mission Impossible Team

Mission Impossible Team, here’s your assignment today. Unfortunately you failed your last assignment: replicating Mann’s claimed correlations. But that probably was impossible. Today your assignment is probably possible, but is a dangerous expedition into the dark underground – into the terrifying world of Mannian RegEM. Courage and perseverance will be required. You may not return alive.

Let’s start with something simple and seemingly innocent – a location map of Loehle’s proxy selections. This shows the locations of 18 proxies in various locations around the world. Three “regional” proxies – two in China and one in North America – are shown at their approximate centroid.

Next here is a location map for the 22 proxies in the Mann et al 2007 network (the network formerly known as MBH98). As with the Loehle network, the two North American PCs are shown in locations near their approximate centers of gravity.) It’s more land-oriented than the Loehle network, perhaps a little more North American-oriented, but it looks innocent enough.

Although Mann et al said (and Zorita et al 2003 unfortunately adopted this point) that it was impossible to allocate weights to the individual proxies, this is untrue. Vast quantities of linear algebra cancel out enabling the NH reconstruction to be expressed as a linear weighting of the individual proxies (some with negative weights.) In the graphic below, I’ve made the area of each dot proportional to the MBH98 weight of the Mann et al 2007 proxy. The visual impression is obviously entirely different. The point here is one that’s been very clear to me for a long time and I’ve tried various ways to convey this message, but I think that this image may finally enable a wider audience to fully understand what the “more sophisticated” (using JEG’s words) algorithm does.

The MBH98 network is essentially a combination of only 4 series: the North American PC1 (which is 95% Graybill bristlecone chronologies), Gasp” – a questionable series, which had an unreported extrapolation to get it into the AD1400 network (this is the sort of unreported accounting adjustment that sets off alarm bells in the real world although not in real climate; Briffa’s Tornetrask series (which was manually altered by Briffa in Briffa et al 1992 – an alteration reported but nonetheless unjustified; and Cook’s Tasmania ring width series. The other series are just nothings under MBH98 weighting. Some negative weighted proxies (e.g. ice core accumulation) have a plausible physical interpretation as inverted. In my opinion, these can be counted as positive correlations and arguably the negative of the proxy should be collated. One should not otherwise invert things like tree ring proxies after the fact.

Now you’ve all heard that Mann can “get” a HS without using PCs. Again I’ve observed in the past that this is really just a trick for using the bristlecones, but this point can be illustrated rather neatly using the graphic technique of the above maps. First here is a map of the locations of all 95 series in the Rutherford et al 2005-Wahl and Ammann 2007-Mann et al 2007 AD1400 network, with the size of the dots set so that the total dot area is equal to the other two cases. You don’t have to be Ethan Hawkes to notice that the no-PC network is predominantly in the U.S.

Now in the world of temperature histories that we’ve been exploring from time to time, we know that the U.S. temperature history in the 20th century has a different course than the ROW; indeed, in the US, according to NASA, 1934 was the warmest year of the century (inquiring minds want to know whether it was the warmest in a millennium or even in a millllll-yun years). But Gavin Schmidt, functioning both as NASA spokesman and as an able real climate spokesman, has told us, as has Hansen, that the US constitutes only 6% of the global land surface and 2% of the surface and is thus unrepresentative. One would presume that the weightings in the no-PC network would downweight the over-populated U.S. proxy network to reflect the lack of geographic balance introduced by abandoning PC networks – even if it meant downweighting the bristlecones.

OK, I was just teasing. I didn’t really expect you to think that they’d downweight the bristlecones. Here’s what the weights work out to in the no-PC network. They are – if anything – even more concentrated in the U.S. southwest (bristlecones) than the unweighted distribution, which was already heavily weighted to the U.S. The main HS-shaped Graybill bristlecone chronologies each strongly impact the final results, so that the early portion of the MBH reconstruction – whether modulated through Mannian PCs or in the no-PC situation – still is essentially the sum of the Graybill bristlecone chronologies. Mann’s incorrect PC method is not the only method of overweighting the bristlecones.

What caused the MM reconstruction to differ was that different weights were assigned to the proxies – with less weight to the bristlecones, other series, especially Tornetrask, Quelccaya and (oddly) Tasmania gained weight. It wasn’t that we believed tha a weighting of these series was specially wonderful as a reconstruction, but that the recon was unstable to bristlecone weighting.

You can immediately see the problems that the Ababneh update of the Sheep Mountain chronology pose for these weightings. The image below (previously posted) compares the Graybill Sheep Mountain chronology (the most heavily weighted series in the MBH98 network) to the Ababneh version. Merely substituting this version for Sheep Mountain in other networks already accounts for change. In this case, there are a few other highly weighted bristlecone chronologies, but all are by Graybill in the 1980s and the replication problems at Sheep Mountain should be setting off alarm bells for the other chronologies. (Of course, there is heavy security on the updated Sheep Mountain measurement data, security which will be too hard for even a Mission Impossible Team.)

Although this network was first used in MBH98, it has been applied subsequently in Rutherford et al 2005, Wahl and Ammann 2007 and most recently Mann et al 2007.

In each new guise, it has become harder and harder to determine what the weights of the individual series are in the final reconstruction. Mann et al 2007 says that it is impossible to know, but Tapio Schneider has observed that, while the algorithm is nonlinear, the regression coefficients can be extracted from the RegEM algorithm.

So, Mission Impossible Team, your task is to determine the weights of the individual proxies in the Mann et al 2007 RegEM version and to plot them on a map such as the above. Your instructor, the brave and resourceful JEG, has unfortunately gone missing, perhaps swallowed up in the swamp of Mannian pseudo-covariances. But before going missing, he left us with the message that the failure to show location maps was pseudo-science. (Yes, he didn’t say that the failure to show a weighted location map was pseudo-science, but he’d probably have agreed that it was a good idea.) So, Team, in memory of JEG, return with the weighted location map for the AD1400 Mann et al 2007 RegEM proxy network. The fate of the planet depends on you.

58 Comments

What happens if temperature is converted to heat, using the average levels of humidity and assuming that the troposphere is only 7 km high at the poles but extends up to 17 km at the equator. Does that have a big difference on the weighing of the proxies?

Steve:
Given the known global variance in anomalies, doesn’t the paucity of the proxies argue for large error bands on any multi-proxy study? Your map just drives this home with the obvious unrepreseantive nature of the sample – independent of the fun and games with the weights.

It’s interesting to see again how very few Southern Hemisphere proxies are used, but then again, the Hockey-stick is a Northern Hemisphere only graph. It would be interesting to have more Southern Hemisphere proxy data, but I guess gathering it is too inconvenient.

Great maps, Steve. Now I can tell people where to see maps when they do not understand what I am saying.
Also – if the southern hemisphere was weighted like the bristlecones – I bet that would flatten the stick a bit.

Of course, we can find in the mud the chemical tracks of the chemical and physical conditions and variability of the environment. That’s the raison d’être of Dr. Loehle’s work. He didn’t ponder the proxies derived from tree-rings, but from isotopes, diatoms, foraminifera, etc., which makes his research consistent and trustworthy.

A principal author on denrochronology in Tasmania has been Brendan Buckley. I have written to him asking if he is willing to provide raw data for further analysis. An abstract of some analysis follows from the 1997 paper

Abstract A network of seven Huon pine ring-width chronologies is constructed from sites ranging in elevation from 200 to 950 metres above sea level in western Tasmania. The chronologies are analysed individually and collectively to explore Huon pines response to climate as a function of elevation. Three chronologies from greater than 700 metres in elevation exhibit a strong, direct response to temperature for most growing season months (p

The Huon pine is particularly important because logs of it buried in streams for thousands of years are in adequate condition to study. However, there does seem to be a problem similar to the Bristlecone Pine, that those few trees that struggle at high altitude are thought to show most temperature relationship, while those at lower, wetter altitudes appear to respond more to diverse climate factors. How one draws the line between the two cases, mathematically and quantitatively, is an interesting problem.

wouldn t it be at least sort of fair, to give these proxies some special treatment? like the bristlecones?

or at least to mention the problem now and then?

Steve: You’ve mentioned it. This really doesn’t have anything to do with Mann weighting which was the thrust of this post. Do you have any problems with this being moved to a Loehle post where it belongs?

Steve: Youve mentioned it. This really doesnt have anything to do with Mann weighting which was the thrust of this post. Do you have any problems with this being moved to a Loehle post where it belongs?

i have no problem with you moving it (and/or erasing this post)

but reading your post and comments, i am left with a feeling that the Loehle proxies are “better distributed” and with “less flaws” than the Mann ones.that is simply false.

My eyes! My eyes! (I still feel ‘as ill as ever’ reading sub-par grammatical execution like this, never mind any *possible* deceiving, mis-directing or ill-intent of the poster; I can only surmise that the goal here being an infamous ‘meddle* of freedom’ award for being voted off-the-island with continued nit-picking diversions that sap the energies (and patience, patience: the ability to endure waiting, delay, or provocation without becoming annoyed or upset) of posters that might otherwise better be directed elsewhere, and I speak as one who wants to look at Wein’s law/Stephan-Boltzman Law/spectral energy distribution in more detail under the influence of … but I digress)

JEG wasnt prepare to deal with real scientist that answer to criticism that quickly. In less than a week all is concern were addressed.

JEG was probably surprised and must have been disappointed that his review was pre-empted by my own equally critical review. But as an Aikido master, I think he was prepared to deal with this. It is not true that all of our concerns have now been addressed. We will need to wait to see what Loehle and JEG come up with in terms of bootstap confidence intervals. Last we heard from JEG he was running some Monte Carlo simulations. I suspect he’s busy. Good science takes time.

I thought people might be interested in this excerpt from a paper by the late Jonn Daly regarding the provenance of the pine tree data from Tasmania.

The data used by Mann is, I believe, Cooks Johnson lake (Mt Read) material that in Mann,s data set starts in AD 900.
As opposed to Buckleys Lake Marilyn set that starts in AD 1058.Is this correct to your knowledge (just to avoid any confusion)?

People might also be interested in where the calibrating temperature records came from, considering that Tasmania is the size of Scotland and Hobart is the second driest city in Australia.

On a personal note; I would like to thank you for this blog. I live in a small country town in Australia and for one reason and another had to finish my formal education at the age of fourteen. The wealth of information presented on this site has kept my brain ticking over furiously for over two years now. Other posters have likened this site to an online university (yes) and that trying to assimilate the information is like trying to drink from a fire hose (I loved that one).
However, I think that in the tradition of a number of scientists that didn’t act in a condecending manner towards “trolls” , you have tried to present science in an accessible manner for the lay person without dumbing things down to the point where polemical points are obscured.
Thank you.

Exhibit 8 – Tasmania, Australia
Tasmania is an island state of Australia, about the size of Maine, deep in southern latitudes. In this exhibit, we not only find confirmation of the Medieval Warm Period, but also obtain some insight into the origins – and the flaws – inherent in the `Hockey Stick’ itself.
Ed Cook, a prominent tree ring researcher, has been a frequent visitor to Tasmania over the past 10 years, taking tree ring samples from a unique species of long-lived softwood known as `Huon Pine’ (Lagarostrobos Franklinii), some of the living trees being over 1,000 years old. Due to Tasmania’s remoteness south of the Australian mainland, Cook’s papers did not receive the critical examination they warranted, as there were flaws in both his handling of local data and in his conclusions.
To calibrate the tree rings against temperature, Cook and his team used urban surface temperature records from the dry eastern half of the island to compare with tree rings taken from the wet western half, even though there were rural surface records in the west from which a more valid comparison could have been made. In his earlier studies, no allowance was given to the Fertilizer Effect of CO2, making his conclusions about recent decades invalid.
Back in 1992, seven years before Mann’s paper appeared, Ed Cook was the co-author of a paper in `Holocene’ [3] in which a time series of Huon Pine tree rings going back to 900 AD was presented. Here is a scan of a graph he presented (colour added for emphasis and clarity).

Fig.10 – Huon Pine tree ring widths from Lake Johnston in western Tasmania
From the above Huon Pine record, it is clear that there were strong growth surges from 940 – 1000 AD and from 1100-1200 AD, during the Medieval Warm Period. Cook acknowledges this fact in his paper.
The Little Ice Age appears weak in this proxy record, attributed by Cook to the moderating influences of the Southern Ocean on such a small island.
The growth spurt of the Huon Pines in the late 20th century (coloured yellow with an identification label added) cannot be attributed to climate alone, but must inevitably result from the CO2 Fertiliser Effect, a phenomenon not allowed for by Cook, but which has since found to be accelerating plant growth all over the world, exactly as predicted by plant biologists. When the late 20th century growth is discounted because of this factor, it is clear that climate was warmer during medieval times in Tasmania than is the case today.
Cook’s drawing of a heavy curved line to act as his `zero’ line which he believes to be largely non-climatic in origin clearly imposes his own subjective view of what the data means. If on the other hand the `general shape of the growth trend’ (as he puts it) is climatic in origin, the whole record would then indicate an even stronger imprint of the Medieval Warm Period.
In the same paper, Cook used that subjective zero line as a basis to reconstruct growing season temperatures in Tasmania, producing a 25-year `low-pass filter’ smoothed graph bearing a striking similarity to the later `Hockey Stick’ produced by Mann. The result of this statistical processing is shown in Fig.11

Fig.11 – Temperature reconstruction from tree rings acc. to Cook [xx]
According to Cook’s explanation as to how he converted the tree ring widths graph of Fig.9 into the temperature reconstruction of Fig.10 (making the Medieval Warm Period all but disappear in the process), he calibrated the growth rings against surface temperatures recorded at three weather stations in Tasmania. He used Hobart (the island’s capital city, pop. 130,000), Launceston (pop. 70,000), and Low Head Lighthouse on the north coast. Hobart has a documented heat island [21], Launceston is similarly affected, while Low Head has a proven local daytime anomaly [4] causing its daytime temperature to rise in recent decades due to vegetation growth close to the instrument creating a mini sun-trap. Upon these faulty records, he developed his whole reconstruction.
A further flaw in his study was the geography of the island itself. Tasmania has two distinct climate regimes – a cool wet climate in the western half of the island, and a dry warmer climate in the eastern half. The sharp contrast between the two is very obvious even to visitors driving across the island (Fig.12).

Fig.12 – Tasmanian climatic zones and locations
The Huon Pines were in the west, close to Mt. Read, in a very high rainfall region, but Cook’s three calibrating temperature records came from the warmer, drier east. While his statistical treatments were elegant and esoteric, the faulty surface records he used invalidate the whole reconstruction exercise.
Clearly, this must also be a fundamental flaw in the `Hockey Stick’ itself, since it too is predominantly based on tree rings, particularly for the first half of the millennium, the rings being calibrated against the northern hemisphere surface record of temperature, a record which is itself severely contaminated by heat islands and other local error effects [4]. A further flaw in such calibration attempts will occur due to the Fertiliser Effect of CO2 enhancing tree ring growth, thus inserting an increasing and structural error into the calibration.

Steve: Thanks for the kind words. I find these matters interesting and am glad that others do as well. I don’t think that one can regard the CO2 fertilization as established. It’s controversial.

I understand your point about CO2 fertilisation and I agree with it.
I wasnt trying to endorse all of Dalys observations, but I certainly was interested in the locations chosen for the calibrating temperature records. Hobart is the second driest city in Australia and is about 100 miles or more away from Mt Read. It has about 3000mm less rainfall per year than Mt Read and is also a very large population centre – Curious – given Dr Mann’s weighting of these proxies.
I also understand that the numbering of the figures is a bit confusing; John Daly kept the numbering from the original authors in the the figures but numbered them differently in the context of his entire essay.
I was very reluctant to tamper with the original document to make the exerpt make more sense and can only suggest that people look at the whole article and make what they will of it. As I say, I was particularly interested in the provenance of the Tasmanian tree ring data that was related to the theme of this thread.

I indicated on another thread that I have quickly realised that the old saying that it is “better to say nothing and let people think you might be an idiot rather than open your mouth and leave no room for doubt” might have some substance. I will however still be lurking for a long time to come.

My tag by the way relates to something that Gould once observed – people struggle with the idea that the world is very old – but the top of Mount Everest is composed of marine shale.

Hi Steve,
the elusive JEG popping in for a minute, after research-hours.
My oh my, you have been posting with Stakhanovist productivity ! So much so that people are right to state i cannot keep up with it, so i will make this brief.

So, Mission Impossible Team, your task is to determine the weights of the individual proxies in the Mann et al 2007 RegEM version and to plot them on a map such as the above. Your instructor, the brave and resourceful JEG, has unfortunately gone missing, perhaps swallowed up in the swamp of Mannian pseudo-covariances. But before going missing, he left us with the message that the failure to show location maps was pseudo-science. (Yes, he didnt say that the failure to show a weighted location map was pseudo-science, but hed probably have agreed that it was a good idea.) So, Team, in memory of JEG, return with the weighted location map for the AD1400 Mann et al 2007 RegEM proxy network. The fate of the planet depends on you.

I mentioned your assignment today to the class, which was surprised you were part of the staff at GaTech. But there are so many things we are ignorant of, that i do hope you will forgive us that one.

You are quite right that i never said “that the failure to show a weighted location map was pseudo-science”, so i wonder what argument you are making there. It’s not the first time you put extraneous words in people’s mouth : is this a proof by numbers ?

I do, however, agree that said maps are a good idea. A very graphic way of representing this set of weights. Congratulations on a useful piece of work.

Now to the “hard” questions you raise : since, in the case of MRWA07, the authors have so diligently made their data available to everyone, and since the RegEM code is public, how “Impossible” is it for a beautiful mind like yours to go figure this out, and plot a similar map next to the aforementioned ?

Why should you ask a class of students – undergraduates and budding graduates – to go do your own homework ? And why do you address them by the derogatory “Team”, as if their soul had already been corrupted by Mannphistopheles you despise so ? Are you denying them the right to come to their own conclusions re: the Hockey Stick debate ?
If so perhaps Prof Cobb and I should stop encouraging them to peruse your pages, which we had heretofore found quite informative.

You know, when i checked CA this morning i felt behind in terms of keeping up with your posts, but after this sort of plaisanterie i feel much less guilty. I was just chastising people on my blog for making McIntyre-bashing comments, praising your many scientific qualities, and how much you bring to the climate debate. I would hate to have to take these words back because you value petty vendettas over scientific investigations.

The real issue is this : i am but an insignificant pawn on this checkerboard, Steve, so if your Audit has as much substance as you contend, then your bright intelligence, tenacity, and flair for writing should destroy my arguments (and those of the entire Team) without recourse to small-change intimidation and cheap jokes. Please don’t let these tendencies over-run what is otherwise an excellent blog that could foster amazing progress in climatology.

Steve: JEG, we enjoy your company here. First of all, I don’t “despise” anyone in this – why would you think that? Don’t project the feelings of others onto me. I disagree with people but I really don’t despite them. At the NAS panel, I made a point of going and chatting to Malcolm Hughes – we discussed pleasantries like a comeback by the Liverpool soccer team. Had either of Mann or Hughes showed up for the NAS panel reception, I would have chatted to them. I chatted with Hegerl and D’Arrigo. At the House hearings, I made of point of going over to say hello to Mann and introducing myself. I saw him on the street at AGU last year and said Hi, Mike, but he walked by stonily. I bought lunch for Caspar Ammann at AGU in 2005. I disagree with them, but I don’t “despise” them. I don’t know that they have quite the same equanimity, but that would be their problem.

You ask the question:

Why should you ask a class of students – undergraduates and budding graduates – to go do your own homework ?

Last time, I looked in the mirror, I was not enrolled in any courses for credit nor have I been for many years. So I don’t have any homework. Am I personally going to try to operate Mann’s code? Not right now. I don’t have Matlab; a couple of CA readers do. They’ve been trying to get it to work and I’ll wait for them. My guess is that the weights will be like MBH98 based on the similarity of the recons. After all, even the RegEM recon is just a linear combination of the proxies. For students that are trying to understand how these algorithms work, I thought that it was rather a good exercise. Sure I wrote it in a lighthearted way, but it is a good exercise for all that.

looks like 1°C increase over 100 years. the end of the curve seems to show a similar increase in 50 years.

#25 Yes, same level rise in shorter time frame, but coming out of the little ice-age. Now put some confidence intervals on those curves, sod. How high might the rate of warming have been from AD900-1000?

Exactly why do you assume your class at GT is the “Mission Impossible Team?” Why don’t you realize that much of the “homework” is rhetorical in nature, hence statements such as, “Unfortunately you failed your last assignment: replicating Manns claimed correlations. But that probably was impossible.”

As much as many posters here have mentioned your name and your class, this blog is not about you.

Steve,
I am happy to hear that you don’t despise any of the Team members. It’s not always obvious from your posts.
Now, i agree with you that if the class was about replicating Mann’s work, yours would be a great suggestion. Last i checked, however, you were not on the teaching staff at GaTech, hence a slight surprise to see you enunciate their assignments.
You have a very good point regarding the excessive weight of certain Colorado trees in the ‘NH’ reconstruction. That weight would be justified if it meant that they are somehow recording NH temperature better than any other proxy – but you and several dendroclimatologists have shown that the divergence effect is marring such records in the late XXth century, so their weight should be accordingly revised.

I regret that Matlab is not as freely available as R. Sadly, this does not make Tapio’s code as “public” as i claimed. Hopefully some CA readers will be able to help you there.
For different reasons, i have a hard time following your arguments sometimes, as R code is like Aramaic to me.
Again, a question of language…

Michael Jankowski ,
“Mission impossible team” was equated to said class EAS8100 (a typo for EAS8001) in the post’s title. Or am i jumping to conclusions ?
Of course this blog should be about auditing Climate and not auditing JEG . My work, yes, go crazy. Other things are simply inappropriate.

#32. A couple of comments from a CA reader on his replication efforts so far with the archived code:

I have the B matrices for both the low and high freq for the end of 1400-1450 period, but the high and low freq proxies don’t add up to the original proxy data. Not sure what the problem is, but Rutherford et all do so much arbitrary rescaling I may have missed one.

Secondly:

the B matrix from the low frequency run has complex numbers in it, while the imputated data doesn’t, which makes no sense. I need to do some trouble shooting. It will be this evening before I get to it.

JEG, you say:

You have a very good point regarding the excessive weight of certain Colorado trees in the NH reconstruction. That weight would be justified if it meant that they are somehow recording NH temperature better than any other proxy – but you and several dendroclimatologists have shown that the divergence effect is marring such records in the late XXth century, so their weight should be accordingly revised.

You’re mixing up a couple of different issues here. The “Divergence Problem” is the failure of ring widths and densities for the majority of tree ring sites that are supposed to be temperature indicators to keep pace with temperature.

The Graybill bristlecone chronologies were the opposite: they had growth increases that were higher than could be accounted for by temperature. California bristlecones are the main ones, Colorado is secondary. I sampled the Colorado ones only because I was visiting my sister in Colorado Springs.

Your position on the Graybill strip bark chronologies is that “their weight should be accordingly revised”. The recommendation of the NAS Panel was that they should be “avoided” i.e. their weight be reduced to zero. What is the basis for your rejection of the NAS panel recommendation or do you agree with it?

Your position on the Graybill strip bark chronologies is that their weight should be accordingly revised. The recommendation of the NAS Panel was that they should be avoided i.e. their weight be reduced to zero. What is the basis for your rejection of the NAS panel recommendation or do you agree with it?

JEG– I’m trying to teach myself R. It is, indeed, obscure! When I read the various guides, I wonder if a person is required to first learn oop (object oriented programming) before dealing with R (aka, ‘the beast’).

In about a month, I’ll probably know whether it’s worth dealing with the learning curve. Meanwhile, the simplest things like trying to read a file or take an average seem horrific.

On the irony & the class assignment. …. the Wall Street Journal recently ran an article on teaching people to understand those from other cultures. It appears that Americans, British and Danes use irony much more often than most Europeans or Asians. The article didn’t mention Canadians, but it appears Steve is someone inclined toward irony. Even if your class didn’t let on, if they are at all bright, they likely suspected irony in the title.

The homework assignment is for those here on CA that are so inclined, I beleieve.

Seems to assign it to the “EAS8100 Mission Impossible Team” is a bit of satire. Steve’s been known to do that before from time to time in his titles, and to have a bit of fun with it in the article. Makes the subjects a bit more interesting I believe. But that’s just me.

I posted this on the wrong thread and while it is not really worthy of a second chance, I am growing impatient with what I see as JEG’s failure to engage as a climate scientist.

JEGs ability to turn smart phrases while verbally tap dancing his way around this blog is most appreciated and an unexpected bonus coming from what one might expect to be a rather matter-of-fact climate scientist. His approach, while smart-alecky in context, does allow one to take a personal broadside and still chuckle at his remarks.

My question remains: after this flamboyant display of JEG, the erudite, and the urbane, will we be privileged with the presence of JEG, the serious climate scientist, here to answer rudimentary questions (and particular those posed by JEG, the viewer of misconceptions by CA participants) like, for example, a simple explanation of how teleconnections have been validly applied to a temperature proxy.

RE 45. Most people who mention wittgenstein in non philosophical conversation have
never read wittengstein. They have read “about” him. They have perhaps heard the term
“language game” and think they understand one of the quirkiest thinkers of the early
2oth century. It’s an indicator of faux erudition.

In about a month, I’ll probably know whether it’s worth dealing with the learning curve. Meanwhile, the simplest things like trying to read a file or take an average seem horrific.

I couldn’t disagree more. You can take an average of a vector x by:

mean(x)

Or you can standardize series by:

scale(x)

What could be simpler?

There are several read functions that I use. All can read directly from internet url’s

read.table (“http://…”, ) is versatile. If you want tab-separated, use sep=”\t” as an option.
read.fwf (url, widths=c(6,8,5) ) will read from fixed format files. fill=TRUE is a useful option in both cases. skip= is also useful.
read.csv is handy

I love the read functions in R.

The only thing that is slightly troublesome are Excel tables but that’s true of anything. I save them locally in tab-separated format and then use read.table( …, sep=”\t”) to retrieve.

I think you will find that JEG has been doing his Monte Carlo experiments in relation to ENSO teleconnection modulations caused by variations in IPO, PDO etc, (very large numbers indeed JEG).
Perhaps he found that there was too much chance, chaos and statistical fluke involved and is looking for a better way.
Can you please correct me if I am wrong JEG.

@steveM–
Well… the files you download off the web must be much more conveniently organized than the one I downloaded to teach myself. I can definitely get the mean values of a vector– or whole column of numbers–, but I always find myself wanting to calculate conditional means and discovering a drop out because a thermometer must have broken. So, then I have to google how to do something to create the vector given the information actually in the file so I can later stuff in command like mean()!

(It’s never hard in the end, but… well.. googling to find out how to do things is not that efficient. Why in the heck did Rproject organize their help pages using frames? Did they want to make it impossible for search engines to send users to the useful pages?!

There are lots of cool things about R, but other programs that permit selecting columns etc. are easier. (Though, ultimately, horribly limited.) Mathematica is better documented for someone who doesn’t want to read extensive discussions about how the package creates objects and that about that things are classes and subclasses or something or other.

I’ve subscribed to the email list, but I clearly need some sort of decent book to flip through. (Preferably one with real examples printed out. )

Do the files you find always have precisely what you want and only what you want in a row or column? I grabbed a file with 83 years worth of daily temps. I’ll eventually want average over full years. (Done that.) Average over individual days of the year. And/ or apply other processes based on these conditions. (Though, I’m not entirely sure what I’ll do in the end.)

Maybe there is a user friendly resource out there, but I can’t say I’ve found it. (Of course I have the introductory manual. It’s not useless, but using that is a bit like trying to learn Italian by reading a dictionary!

A good book book with some title like “Learning R:” would be nice. I guess I should check Amazon.com! :)

I am also learning R and have found Using R for Introductory Statistics by John Verzani and The R Book by Michael J. Crawley to be most useful. One, I have to relearn statistics while learning R. If you have the statistics down, then the The R Book will be big help. I have found Time Series Analysis and it Applications with R Examples by Robert Shumway and Davis Stoffer helpful also, as they include climate and solar time series examples. Hope this helps.

Lucia,
R is not obsucure nor requires object oriented programming (what’s that ;) ) knowledge ! R is matrix based just like Matlab. R is supported by many many manuals, books and dedicated forums.
You cant’ have simpler and more powerfull.