Gilberto Titericz Q&A

How did you start with Kaggle competitions?

I am an electronic engineer, but I always had interest in machine learning algorithms. In 2012 I attended the Google AI challenge and when it finished I looked for another way of competing online and found Kaggle via a Google search and I loved it. My first competition was Global Energy Forecasting Competition 2012 - Wind Forecasting and I luckily placed 3rd using a bag of Matlab neural networks. Since then, I've tried to learn more from each competition.

What is your first plan of action when working on a new competition?

First of all it is very important understand the problem, the features, and the evaluation metric. After that I look for a good cross validation strategy. Those are the very basic and essential steps. Only after that you can start thinking about feature engineering and training algorithms that match the problem.

What does your iteration cycle look like?

I always prepare the dataset and apply feature engineering as much as I can, then I choose a training algorithm and optimize hyperparameters based on a cross validation score. If a model is good and stable I save the trainset and testset predictions. Then I start all over again using another training algorithm or model. When I have a handful of good model predictions, I start ensembling at the second level of training.

What are your favorite machine learning algorithms?

Gradient Boosting Machines are the best! It is amazing how GBM can deal with data at a high level of depth. And some details in an algorithm can lead to a very good generalization. GBM is great dealing with linear and non-linear features, also it can handle dense or sparse data. So it's a very good algorithm to understand core parameters and it's always a good choice if you are not sure what algorithm to use. Before I knew of GBM, I was a big fan of neural networks.

What are your favorite machine learning libraries?

I have many favourite libraries, but I will try to rank them according the ones I use most:
- XGBoost - Fast and optimized GBM implementation.
- Vowpal Wabbit - Very fast with lots of parameters and linear algorithms.
- LibFM - Factorization Machines.
- scikit-learn - Lots of functions and algorithms.
- Lasagne - Very good and fast NN implementation (but unfortunately I don't have a GPU).
- Matlab Neural Network Toolbox - Classic.
- R GBM, randomForest and glm - Classic.

What is your approach to hyper-tuning parameters?

I tried grid search in the past, but I found that it consumes a lot of time. So I often use manual brute force searching based on my previous experience. And I always use cross validation to consistently improve the evaluation scores.

What is your approach to solid CV/final submission selection and LB fit?

I always trust my local cross validation score, but in some competitions when the public testset has enough samples and the data is not very noisy I will trust both CV and LB. A good scoring formula I've used before is:

In a few words: What wins competitions?

What is your favourite Kaggle competition and why?

Otto is my favourite. Not only because I won, but in this competition it was possible to apply various methods of machine learning and ensembling techniques. In addition, it was the biggest competition in terms of number of competitors in Kaggle's history.

Final private leaderboard for the Otto Product Classification Challenge

What was your least favourite Kaggle competition experience?

Competitions with very small trainset datasets. It makes cross validation and final results unstable. Now I just avoid noisy and small datasets, because it's very hard to apply feature engineering and cross validation efficiently.

What field in machine learning are you most excited about?

Nowadays I'm very interested in deep learning. I think it can still surprise us in the future.

But I'm not an expert in the subject matter of all of my winning competitions. Some examples are: Otto Group, Caterpillar, Fast Iron, AMS 2013-2014 Solar Energy Prediction Contest, Springleaf and GEFCom2012. So I believe for most of the problems you can do well even if you aren't an expert.

What do you consider your most creative trick/find/approach?

I don't have that creative trick, but if I can say what I use that gives me more spots it is a second training level ensemble technique. Some people also call that: "stacking models", but it's all the same. That technique depends a lot of correct cross validation and generation of non over fitted meta features. Also it can increase a lot the performance if meta features are generated using different features and training algorithms, because L2 can extract only the best predictions of each meta feature.

How are you currently using data science at work and does competing on Kaggle help with this?

I work as an automation engineer in a oil refinery. So I don't have much chance to use data science for it. But I can say I'm looking foward to change it on next year. And I can say that most of what I know today about data science I owe it to Kaggle.

What is your opinion on the trade-off between high model complexity and training/test runtime?

That a good question. I confess that I just care about training runtime. Model complexity must be proportional to the CPU power you have and the perfomance that you want. So I always start using simplest dataset as possible and take notes of its training time and performance. After that a good approach in to increase model complexity and note if it increase performance and times. A good tradeoff between complexity and time depends a lot of how much time you have to spend increasing complexity of model and waiting the training time. Usually models that you can build and train in just a few hours are the best, because you will need time to fine tune parameters and it's normal having to train it a lot of time.

What is your view on automating Machine Learning?

Automation is good for preprocess some dataset conditions, cross validation folds, model selection, feature selection and hyperparameters search.
I think all other processes are better if have human interference.

What did you learn from doing Kaggle competitions?

Kaggle in an excellent place to learn about Data Science. It offers many kind of datasets covering a vast type of problems. I learned a lot after those three years competing and I can say it is addictive. Specially if you are that type of person that like to see the machine learning magic in action.

Bio

I have a MS in Electronic Engineering. In the past 16 years I've been working as an Electronic Engineer for big multinationals like Siemens, Nokia, and later as an Automation Engineer for Petrobras Brasil. In 2008 I started learning Data Science by myself, at first to improve personal stock market gains. In 2012 I joined Kaggle.com and started to continually learn more. See Gilberto's LinkedIn.