Buell: ‘Castles in the Clouds’

CA reader hfl, who cited Buell’s documentation of the dependence of principal component patterns on shapes, has sent me a scanned pdf version now available here. It concludes by observing that analyses that fail to consider this phenomenon (and there is ample evidence that Steig et al falls into this category) “may well be scientific level with the observations of children who see castles in the clouds”. [Update: Here is a rendering by a CA reader (now using Lucy Skywalker hockeystick version)]

Here is hfl’s summary:

Forgive me if this has been discussed in past threads on PCA (although I couldn’t find it in a quick search of the site), but it’s worth noting that the issue of principal component pattern dependence on domain shape is (or was) well known within the atmospheric sciences community. It was first documented by C. Eugene Buell in a 1975 paper published in the Proceedings of the Fourth Conference on Probability and Statistics in Atmospheric Sciences. Buell’s work focused on square/rectangular domains, particularly because the latter approximated the shape of the conterminous U.S. This was followed by a second paper at the Sixth P&S Conference in 1979 which states: “When a region with well defined boundaries is concerned, the EOF’s computed over this region are expected to be very strongly influenced by the geometrical shape of the region and to a large extent independent of where the region is located. As a consequence, the interpretation of the topography of the EOF’s in terms of geographical area and associated meteorological phenomena should be looked on with suspicion unless the influence of the effect of the shape of the region has been completely accounted for. Otherwise, such interpretations may well be on a scientific level with the observations of children who see castles in the clouds.”

Buell’s work generated considerable discussion within the atmospheric sciences literature because PCA (or, as they referred to it, EOF [empirical orthogonal function]) analysis was in widespread use. Mike Richman, in a paper published in the Journal of Climatology (1986) entitled “Rotation of Principal Components” made the case that component rotation appears to eliminate the problem of domain shape dependence. Richman’s paper is worth reading in that it reviews and considers nearly all of the important work that had been done using PCA on meteorological and climatological data up to that time, including work by Gerry North and others familiar to CA readers. So these problems have been well documented and understood for a long time. Like so many other elements of meteorology and climatology, though, modern climate science appears to have forgotten some important statistical insights produced by it’s own practitioners . . . indeed, some of the practitioners themselves appear to have forgotten what they wrote.

51 Comments

With all due respect, I find the “castles in the clouds” statement somewhat funnier than “done on the back of an envelope.”

It’s interesting to see older papers like this with hand-drawn graphs compared to modern equivalents with computer-generated plots which are simultaneously more accurate (not to imply scientific accuracy or fitness) and aesthetically pleasing. Science sure has changed.

The doesn’t contain anything as poetic as “castles in the clouds”, but here’s the full paper on the problem as recognized in population genetics. Instead it uses phrases like “mathematical artifacts” and “providing a’null’ expectation against which observed PCs may be compared and contrasted”.

Thank you! I suppose you could have some smoke rising near the MMTS and leave the BBQ to our imagination. Unfortunately there don’t seem to be any windows nearby to have an air conditioner in. The arm on the flag does have a nice hockey stick shape, though, I notice.

If this shape dependency holds, how is it accounted for in the determination of the AAO which is part of Steig et al’s finding? see link and associated method. Surely this must be accounted for in the generation of the AAO?

MichaelJ, nice paper. I particularly enjoyed this part (emphasis mine):

In summary, we have shown that (i) when analyzing spatial data,
PCA produces highly structured results relating to sinusoidal functions
of increasing frequency; and (ii) insofar as PCA results depend on the
details of a particular dataset, they are affected by factors in addition to
population structure, including distribution of sampling locations and
amounts of data. Both these features limit the utility of PCA for
drawing inferences about underlying processes, a fact previously noted
in climatology (for example, ref. 17).

Recent research has pointed to a number of inherent disadvantages of unrotated principal components and empirical orthogonal functions when these techniques are used to depict individual modes of variation of data matrices in exploratory analyses. The various pitfalls are outlined and illustrated with an alternative method introduced to minimize these problems via available linear transformations known as simple structure rotations. The rationale and theory behind simple structure rotation and Procrustes target rotation is examined in the context of meteorological/climatological applications. This includes a discussion of the six unique ways to decompose a rotated data set in order to maximize the physical interpretability of the rotated results.

The various analytic simple structure rotations available are compared by a Monte Carlo simulation, which is a modification of a similar technique developed by Tucker (1983), revealing that the DAPPFR and Promax k = 2 rotations are the most accurate in recovering the input structure of the modes of variation over a wide range of conditions. Additionally, these results allow the investigator the opportunity to check the accuracy of the unrotated or rotated solution for specific types of data. This is important because, in the past, the decision of whether or not to apply a specific rotation has been a blind decision. In response to this, a methodology is presented herein by which the researcher can assess the degree of simple structure embedded within any meteorological data set and then apply known information about the data to the Monte Carlo results to optimize the likelihood of achieving physically meaningful results from a principal component analysis.

So the problem has been recognized for over twenty years … but in the meantime, the Team has moved on. Kinda reminds me of the Steve Martin movie where he’s in a restaurant and says something like “Hey, you can’t fool me. This wine is twenty years old! Bring me some of the new stuff!”

No slight to hfl in the OP, but Richman (1986) has been mentioned before – in the context of Wilson’s dendro work:http://www.climateaudit.org/?p=1285
It is also cited in the paper mentioned by MichaelJ in #12 and it contains the start of the answer to bernie’s #12. By finding a “suitable” rotation you emphasize “simple structure” and your eigenvectors move away from being a parody parade of harmonics. That’s the theory. Read it.

Re: bender (#16), Nice tip. Personally, and rightly or wrongly, I have always used a rotation to make sense of a factor. With survey data it is very difficult to interpret the factors any other way because the definition of the factors has to somehow reflect the questions that were asked. (We are plagued however by our own forms of autocorrelation such as leniency or halo).

The world is a complex place and climate factors have been known to affect different geographical cofigurations (high altitude-low altitude, continental coast-interior) in different ways. Rotation of axes can, in some cases emphsize those relationships.

Look at a simple (non-climate) example. Suppose you graph a bunch of peoples’ heights versus their weights. Assume that you scaled them in such a way that most of the subjects fall close to a 45 degree line in the graph going through the origin. Now rotate the coordinates 45 degrees retaining orthogonality of the axes. It is not inconceivable that the new axes represent somewhat different factors than just height and weight. One could label one axis as “size” which differentiates larger and smaller persons, and the other might be viewed as “body shape” (e.g. taller thinner persons or heavier and shorter persons) in the population from which you took the subjects.

It is not inconceivable that similar considerations could apply here. However, without reasonable statistical considerations AND demonstrated physical (non-teleconnective) causes, it would be no more than arm-waving and pointing at the castles in the skies.

As a side note. when I was teaching, I used to call it “horsies in the clouds”. I suppose that my students weren’t as mature as as the crowd inhabiting CA…

So will Prof. Steig reveal his data and confirm or collapse your conjectures, or will he withdraw the paper and conceal his castle in clouds forever? Decisions. Decisions.
==============================================================

Domain shape dependence
The predictable patterns which often occur for PCs in climatological studies are due mainly to the spatial correlation usually present in such data, with domain shape acting as a significant, but secondary, factor.

It is evident from these examples that the shape of the boundary of the data points to which the correlation coefficients (or covariances) of the matrix problem belong is a dominant factor in determining the topography of the proper functions (EOFs) of its solution.

Reflecting a little on Buell and Richman, I think that there are some important aspects of the relationship that have been reported here than are not present in the earlier literature.

I haven’t parsed the articles in detail and do not exclude the possibility that I’ve missed something relevant, but in a first read, I don’t see that either article identified the form of the patterns as clearly as we’ve pinned down here. We’ve put a functional form on the eigenfunctions, relating them to Chladni patterns (and other homomorphic structures). I didn’t see that in the earlier literature.

Given the present (more informed??) interpretation of these patterns as Chladni-type patterns, I’m not at all sure that rotation of principal components is an omnibus cure, so the specific recipes in Richman would need to be parsed. I would place more weight on the recognition of the problem, than the prescription.

bender, as to the bolded item in Jolliffe’s 1987 comment, I doubt that Jolliffe would preclude the possibility of domain structure being dominant in some circumstances or that he would disassociate himself from the statistical requirement that the authors show that their eigenvectors are not mere castles in the clouds (as in the nice Richman quote that you made on this topic.)

Steve:
If the pattern of the non-rotated PCs are essentially associated with the spatial autocorrelation, in what ways would a rotation actually reveal some underlying structure reflective of physical processes as opposed to spatial relationship. Shouldn’t one or more of the rotated factors actually reflect the spatial relationships? For example, if all that was there was the spatial relationship then they should appear as the rotated factors? But I am treading at the edge of my current ability in this arena – which may be obvious by my questions.

RomanM:
That’s how I recall it. I guess the question I have is the extent to which the AAO is a statistical artifact. Given the horsepower available, it is hard for me to imagine that those responsible are not aware of these spatial correlation issues.

Re: bernie (#35),
Of course they are aware of it. The question is why, given the known issues, do they make claims that they don’t bother to prove. And I think the answer is simply Nature’s ultrastrict word limit. Nature has no business fast-tracking speculative climatology papers. Save that for the specialist climatology literature.
.
The story here is not so much what Steig et al did inside the paper. It’s what the Nature reviewers didn’t do, and, even more, what the Nature media people did to spin the story. Steig et al. went along for the ride. And why not?

Reflecting a little on Buell and Richman, I think that there are some important aspects of the relationship that have been reported here than are not present in the earlier literature.

I haven’t parsed the articles in detail and do not exclude the possibility that I’ve missed something relevant, but in a first read, I don’t see that either article identified the form of the patterns as clearly as we’ve pinned down here. We’ve put a functional form on the eigenfunctions, relating them to Chladni patterns (and other homomorphic structures). I didn’t see that in the earlier literature.

A CA reader writes offline:

Steve,

Have a look at the literature on fluid dynamics.

Turbulence, Coherent Structures, Dynamical Systems and Symmetry

Holmes was a great teacher. I seem to remember that the eigenfunctions of homogeneous turbulence can
be shown to be sines and cosines. This is intuitive because homogeniety implies translational invariance:
you move from point to point and everything looks the same. Sines and Cosines (or more formally complex exponentials)
are translationally invariant eigenfunctions. Translating just introduces a constant phase factor Psi(x)=exp(I*k*x)
Psi(x+dx)=Exp(i*k*dx)*Exp(I*k*x)

In other words the eigenfunctions for KL/PCA are determined by the symmetry properties of the system under decomposition.

Re: Jeff Alberts (#38), (and for anyone interested in header graphics) — it’s best when the header can seamlessly repeat (one way or another). People have different screen widths; it’s nice when the header “works” for everyone.

I agree, MrPete. However, I was using Steve’s current header graphic as a template for width and height (copied the image and used the raw size as opposed to the visible size in the browser). The width is greater than 1600 pixels, so while you may have a screen res wider than that, is it likely that you’d have your browser taking up the entire width? A lot of sites also have fixed width content areas with fixed width headers, so it’s not a given by any means. Anyway, it was a quick and dirty…

I did a couple of videos of the satellite temperature trend because I wanted to see how PC3 changed the temp anomaly. It is visible in the anomaly video, you can see the oscillations back and forth along PC2 and up and down from PC1 and a sudden left-right motion in PC3. I then did a simple enhancement to make it more visible, what I didn’t expect to find was that the actual satellite data from Steig’s conclusion has a temperature reversal from the eastern rim temps to the center of the antarctic.

When I see a semicircle of temps different from the interior it bothers me a bit. The ocean certainly would dampen trends, I’ve lived on both sides of lake Michigan for years at a time. On the west, there’s not usually much effect but on the east wow! It mau be possible with circular weather patterns that the edge of the antarctic would all have similar temps, I’m not sure. I’m guessing the muted temparature half the way around the antarctic could actually exist in the satellite data overlaying the surface stations. This could cause regem to weight the interior with a heavier or even inverted trend from what the surface stations show (e.g. if the trend in the interior is stronger in the sat data, trends at the coast will be amplified in the interior. Since these surfacestations by my above assumptions are therefore in a high gradient area, small adjustments of even things like the 50Km gridsize or .5 degrees of latitude could make a real difference in result.

Either way, it’s interesting to see the PC’s at work. I think the reason Steig chose not to describe the physical meaning of the 3rd PC may be because it doesn’t exist pre-1982. Without the sat data it’s just speculation of course.

Alan Stern, NASA’s “hard-charging” and “reform-minded” Associate Administrator for the Science Mission Directorate, resigned on March 25, 2008,[44] to be effective April 11, after he ordered funding cuts to the Mars rovers and Mars Odyssey that were overturned by NASA Administrator Michael D. Griffin. The cuts were intended to offset cost overruns for the Mars Science Laboratory. Stern, who served for nearly a year and has been credited with making “significant changes that have helped restore the importance of science in NASA’s mission.”, says he left to avoid cutting healthy programs and basic research in favor of politically sensitive projects. Griffin favors cutting “less popular parts” of the budget, including basic research, and Stern’s refusal to do so led to his resignation.

In response to the internal review, policies at NASA would be changed in a variety of ways: Flight surgeons would be present during the pre-mission suit-up activities, flight surgeons would receive additional training in psychiatric evaluation, and although there was an unofficial code of conduct in place, an official “Code of Conduct” would be written up for employees NASA