I'm a health actuary with Milliman, Inc. I do some traditional services like pricing and reserving, but I also focus on applied statistics and statistical graphing. I have worked with EHR data for some client projects before. Neil Schneider and I have teamed up on many of the Predictive Modeling contests on Kaggle over the last couple years. You'll find us consistently just outside the money and loitering in the forums.

What made you decide to enter?

I enjoy making visualizations. I don't have a lot of experience with data-heavy dashboards, but I constantly make graphs to tell stories. As a part of any Predictive Modeling contest I enter, I create mountains of visualizations. I don't really believe a result until I can see the result. Given this, I enjoyed the novelty of a contest just about visualization.

What preprocessing methods did you use to study the data?

Mostly I dealt with flattening the data to one observation per patient. I focused on Age/Gender, Diagnosis, Medications, Smoking Status and some of the Biometrics. For the Diagnosis and Medication prevalence I did some Bayesian shrinkage since some patients were seen so rarely I did not believe their complete health status had been captured. I used non-linear expansions of the continuous variables since linear assumptions are tenuous at best with real data. My entry goes into more details about the particular form of dimensionality reduction I applied next. Lastly, I did a few supervised learning steps to make the Data.gov mash-up.

How did you decide what aspects of the data to use?

I focused on the portions that could tell a good story of redundancy. I also avoided the portions that would require heavy imputation. I initially chased down physician specialty, but there was too little variation to be interesting in an unsupervised environment. For the Data.gov portion I focused mainly on state-level information since that was the most likely shared key. EHR data is commonly sparse in socioeconomic information, so I chose income data.

Were you surprised by any of your insights or any key features?

I was surprised at how strong the Smoking Status was in the principal component analysis. I expected Age/Gender to drive the major data divisions, but Smoking Status was as important if not more so. Smokers just utilize a very different set of services than non-smokers.

Which tools did you use?

I did all of my analysis in R (http://cran.r-project.org/). I cite all of the great packages I used in my report. It was the first time I had tried flattening a relational database via R, and it went much smoother than I expected. It was also the first time using the Markdown language, which was also surprisingly easy.

What have you taken away from this analysis

A much better understanding of the Markdown language and the RStudio /Knitr implementation in particular. I will definitely be using that in my consulting work going forward.