Poke my eyes out

Perhaps your immediate response is: “I’d rather poke my eyes out with a burning stick than do data analysis.”

There’s a completely different reaction from a lot of people who have experienced data analysis.

Music

It’s not entirely clear why humans like music so much. Part of it may be the guessing game we do. We perceive a pattern in the music and guess where it will go next. One of two things happens:

we are gratified to be right

we are surprised to be wrong

We like being surprised and we like being right, and we like the tension of not knowing which it will be. We look for patterns, and patterns within patterns. That experience of listening to a new piece of music also describes analyzing a new dataset. We create an image of what the data are like; we learn how we are right and how we are surprised. We then form a new image and dive in deeper.

We look for patterns, and patterns within patterns.

R is the grand piano

Twisting the metaphor beyond recognition, there’s a data analysis instrument that is outstanding at making music. It is called R. There are some technical reasons why R is good. There are also social reasons:

it is the lingua franca of statistics

it is rapidly growing in applied data analysis

there are thousands of contributed packages (as of this writing 4953 in the main repository)

Figure 3: Ugly and uninformative. You can also create exceptionally pretty pictures in R, like the Facebook friendship graph (Figure 4).

Figure 4: Strength of Facebook friendship by location. Another possibility in R is to create something beautiful and deeply informative, like the death rate plot (Figure 5) courtesy of Rob Hyndman.

Figure 5: The male to female ratio of death rates in Australia from 1921 (red) through the rainbow to 2009 (purple). If you were hasty enough to poke your eyes out before you learned that data analysis can be fun, don’t despair. You can also analyze data in R with sound:

Snow

It’s a thrill to discover something that no one else knows. Data analysis is one of the surest routes to that feeling. It’s like walking through fresh snow that no other creature has touched. There is a moment to savor after you’ve found something and before others know.

Related Posts

— That experience of listening to a new piece of music also describes analyzing a new dataset. We create an image of what the data are like; we learn how we are right and how we are surprised.

And, more often than not in some areas of data analysis, the surprise is suppressed, since the Daddy Warbucks paying us doesn’t want the data to contradict decided policy. No surprises.

— We then form a new image and dive in deeper.

Too often, a mad scramble to the beach. See the “Timely Portfolio” post on backtesting from yesterday, just a couple of slots down the R-bloggers’ page today.

To the extent that R is being co-opted by partisans (and I’ve not been an R zealot long enough to say “remember when R was used only be objective academics”, or even if such is a true statement), we need be mindful of “figures don’t lie, but liars figure”. To the extent that R enables ever more clever lying by the marginally trained, we need be vigilant. The London Whale, and other such fiascoes, were perpetrated by the unknowing using Excel, the financial community’s tool of choice. At least with Excel the unknowing concocted the messes themselves; as the paper cited by TP makes clear, the quants can concoct opaque, simple answers to complex questions using stat packages (and not just R, to be clear). And, thus unleash a pod of whales on the community. Pandora looks like a fairy godmother by comparison.

One of the most scary description of music, not to talk about the analogy between music and R….come on Music is magic and pleasure, trying to make any comparison with data analysis is just cheating, like when trying to convince a kid to eat something bad…data analysis is a necessity but not a pleasure