Random Forest of 'Give Me Some Credit' Survey Results

The hosts of Give Me Some Credit conducted a post-contest survey and have written a white paper (now available here) on the results. Their predictive modeling of competitor performance confirms many of our intuitions on the wide range of skills needed to become a top Kaggle competitor, and de-emphasizes the importance of domain knowledge relative to data science skills. Here are a few of the high-lights. (Credit goes to Dhruv Sharma for all the graphics)

What different modeling techniques did you try to use? What was your final choice?

Which modeling techniques gave you the most improvement?

Which modeling techniques were the least useful?

The host then ran a random forest of the survey answers to predict performance. Education and years of experience in the credit industry had surprisingly low variable importance for predicting performance. The algorithm used most frequently in the credit scoring industry, logistic regression, performed the worst.

Variable Importance in Random Forest

The biggest predictor of success of top ranking groups was the use of multiple and hybrid models, and top ranking teams tended to use random forests, gradient boosting machines, logistic regression, decision trees which are the basis of both random forests and gradient boosting machines, and ensembled solutions.

The top performers had high proficiency in predictive modeling and less experience in the domain of credit scoring and risk. Credit scoring proficiency and domain knowledge resulted in better performance but only in instances of a great deal of experience (>10 years in credit domain), and high proficiency in credit scoring and predictive modeling. In terms of occupation, computer science and predictive modeling practitioners did the best.

Robert QS

Thanks, very interesting.

Jason Tigg

Great article, shame the axis labels are so hard to read.

Margit Zwemer

Some improvement on the readability.

Jason Tigg

Great thanks!

Kevin

In the first sentence you mention a white paper, is there a link to that somewhere?

Margit Zwemer

I miswrote, paper has not yet been published, still in draft stages. I'll check with the author to see when it will be publically released.

addhyan

Please provide link to the white paper.

Glider

please see previous comment. this was a teaser post, full paper isn't released yet