In this end-to-end Python machine learning tutorial, you’ll learn how to use Scikit-Learn to build and tune a supervised learning model!

We’ll be training and tuning a random forest for wine quality (as judged by wine snobs experts) based on traits like acidity, residual sugar, and alcohol concentration.

Before we start, we should state that this guide is meant for beginners who are interested in applied machine learning.

Our goal is introduce you to one of the most flexible and useful libraries for machine learning in Python. We’ll skip the theory and math in this tutorial, but we’ll still recommend great resources for learning those.

Scikit-Learn, also known as sklearn, is Python’s premier general-purpose machine learning library. While you’ll find other packages that do better at certain tasks, Scikit-Learn’s versatility makes it the best starting place for most ML problems.

It’s also a fantastic library for beginners because it offers a high-level interface for many tasks (e.g. preprocessing data, cross-validation, etc.). This allows you to better practice the entire machine learning workflow and understand the big picture.

This is not a complete course on machine learning. Machine learning requires the practitioner to make dozens of decisions throughout the entire modeling process, and we won’t cover all of those nuances.

Instead, this is a tutorial that will take you from zero to your first Python machine learning model with as little headache as possible!

If you’re interested in mastering the theory behind machine learning, then we recommend our free guide:

Now it's time to start importing functions for machine learning. The first one will be the train_test_split() function from the model_selection module. As its name implies, this module contains many utilities that will help us choose between models.

Next, let's import the families of models we'll need... wait, did you just say "families?"

What's the difference between model "families" and actual models?

A "family" of models are broad types of models, such as random forests, SVM's, linear regression models, etc. Within each family of models, you'll get an actual model after you fit and tune its parameters to the data.

*Tip: Don't worry too much about this for now... It will make more sense once we get to Step 7.

We can import the random forest family like so:

Import random forest model

Python

1

fromsklearn.ensemble importRandomForestRegressor

For the scope of this tutorial, we'll only focus on training a random forest and tuning its parameters. We'll have another detailed tutorial for how to choose between model families.

For now, let's move on to importing the tools to help us perform cross-validation.

Import cross-validation pipeline

Python

1

2

fromsklearn.pipeline importmake_pipeline

fromsklearn.model_selection importGridSearchCV

Next, let's import some metrics we can use to evaluate our model performance later.

Import evaluation metrics

Python

1

fromsklearn.metrics importmean_squared_error,r2_score

And finally, we'll import a way to persist our model for future use.

Import module for saving scikit-learn models

Python

1

fromsklearn.externals importjoblib

Joblib is an alternative to Python's pickle package, and we'll use it because it's more efficient for storing large numpy arrays.

Phew! That was a lot. Don't worry, we'll cover each function in detail once we get to it. Let's first take a quick sip of wine and toast to our progress... cheers!

Step 3: Load red wine data.

Alright, now we're ready to load our data set. The Pandas library that we imported is loaded with a whole suite of helpful import/output tools.

You can confirm that the scaled dataset is indeed centered at zero, with unit variance:

Python

1

2

3

4

5

printX_train_scaled.mean(axis=0)

# [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]

printX_train_scaled.std(axis=0)

# [ 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]

Great, but why did we say that we won't use this code?

The reason is that we won't be able to perform the exact same transformation on the test set.

Sure, we can still scale the test set separately, but we won't be using the same means and standard deviations as we used to transform the training set.

In other words, that means it wouldn't be a fair representation of how the model pipeline, include the preprocessing steps, would perform on brand new data.

Now, here's the preprocessing code we will use...

So instead of directly invoking the scale function, we'll be using a feature in Scikit-Learn called the Transformer API. The Transformer API allows you to "fit" a preprocessing step using the training data the same way you'd fit a model...

...and then use the same transformation on future data sets!

Here's what that process looks like:

Fit the transformer on the training set (saving the means and standard deviations)

Apply the transformer to the training set (scaling the training data)

Apply the transformer to the test set (using the same means and standard deviations)

This makes your final estimate of model performance more realistic, and it allows to insert your preprocessing steps into a cross-validation pipeline (more on this in Step 7).

Here's how you do it:

Fitting the Transformer API

Python

1

scaler=preprocessing.StandardScaler().fit(X_train)

Now, the scaler object has the saved means and standard deviations for each feature in the training set.

Let's confirm that worked:

Applying transformer to training data

Python

1

2

3

4

5

6

7

X_train_scaled=scaler.transform(X_train)

printX_train_scaled.mean(axis=0)

# [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]

printX_train_scaled.std(axis=0)

# [ 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]

Note how we're taking the scaler object and using it to transform the training set. Later, we can transform the test set using the exact same means and standard deviations used to transform the training set:

Notice how the scaled features in the test set are not perfectly centered at zero with unit variance! This is exactly what we'd expect, as we're transforming the test set using the means from the training set, not from the test set itself.

In practice, when we set up the cross-validation pipeline, we won't even need to manually fit the Transformer API. Instead, we'll simply declare the class object, like so:

Pipeline with preprocessing and model

Python

1

2

pipeline=make_pipeline(preprocessing.StandardScaler(),

RandomForestRegressor(n_estimators=100))

This is exactly what it looks like: a modeling pipeline that first transforms the data using StandardScaler() and then fits a model using a random forest regressor.

Step 6: Declare hyperparameters to tune.

Now it's time to consider the hyperparameters that we'll want to tune for our model.

WTF are hyperparameters?

There are two types of parameters we need to worry about: model parameters and hyperparameters. Models parameters can be learned directly from the data (i.e. regression coefficients), while hyperparameters cannot.

Hyperparameters express "higher-level" structural information about the model, and they are typically set before training the model.

Example: random forest hyperparameters.

As an example, let's take our random forest for regression:

Within each decision tree, the computer can empirically decide where to create branches based on either mean-squared-error (MSE) or mean-absolute-error (MAE). Therefore, the actual branch locations are model parameters.

However, the algorithm does not know which of the two criteria, MSE or MAE, that it should use. The algorithm also cannot decide how many trees to include in the forest. These are examples of hyperparameters that the user must set.

We can list the tunable hyperparameters like so:

List tunable hyperparameters

Python

1

2

3

4

5

6

7

printpipeline.get_params()

# ...

# 'randomforestregressor__criterion': 'mse',

# 'randomforestregressor__max_depth': None,

# 'randomforestregressor__max_features': 'auto',

# 'randomforestregressor__max_leaf_nodes': None,

# ...

You can also find a list of all the parameters on the RandomForestRegressor documentation page. Just note that when it's tuned through a pipeline, you'll need to prepend
randomforestregressor__ before the parameter name, like in the code above.

Now, let's declare the hyperparameters we want to tune through cross-validation.

As you can see, the format should be a Python dictionary (data structure for key-value pairs) where keys are the hyperparameter names and values are lists of settings to try. The options for parameter values can be found on the documentation page.

Step 7: Tune model using a cross-validation pipeline.

Now we're almost ready to dive into fitting our models. But first, we need to spend some time talking about cross-validation.

This is one of the most important skills in all of machine learning because it helps you maximize model performance while reducing the chance of overfitting.

WTF is cross-validation (CV)?

Cross-validation is a process for reliably estimating the performance of a method for building a model by training and evaluating your model multiple times using the same method.

Practically, that "method" is simply a set of hyperparameters in this context.

These are the steps for CV:

Split your data into k equal parts, or "folds" (typically k=10).

Train your model on k-1 folds (e.g. the first 9 folds).

Evaluate it on the remaining "hold-out" fold (e.g. the 10th fold).

Perform steps (2) and (3) k times, each time holding out a different fold.

Aggregate the performance across all k folds. This is your performance metric.

K-Fold Cross-Validation diagram, courtesy of Wikipedia

Why is cross-validation important in machine learning?

Let's say you want to train a random forest regressor. One of the hyperparameters you must tune is the maximum depth allowed for each decision tree in your forest.

How can you decide?

That's where cross-validation comes in. Using only your training set, you can use CV to evaluate different hyperparameters and estimate their effectiveness.

This allows you to keep your test set "untainted" and save it for a true hold-out evaluation when you're finally ready to select a model.

For example, you can use CV to tune a random forest model, a linear regression model, and a k-nearest neighbors model, using only the training set. Then, you still have the untainted test set to make your final selection between the model families!

So WTF is a cross-validation "pipeline?"

The best practice when performing CV is to include your data preprocessing steps inside the cross-validation loop. This prevents accidentally tainting your training folds with influential data from your test fold.

Here's how the CV pipeline looks after including preprocessing steps:

Split your data into k equal parts, or "folds" (typically k=10).

Preprocess k-1 training folds.

Train your model on the same k-1 folds.

Preprocess the hold-out fold using the same transformations from step (2).

Evaluate your model on the same hold-out fold.

Perform steps (2) - (5)k times, each time holding out a different fold.

Aggregate the performance across all k folds. This is your performance metric.

Interestingly, it looks like the default parameters win out for this data set.

*Tip: It turns out that in practice, random forests don't actually require a lot of tuning. They tend to work pretty well out-of-the-box with a reasonable number of trees. Even so, these same steps can be used when building any type of supervised learning model.

Step 8: Refit on the entire training set.

After you've tuned your hyperparameters appropriately using cross-validation, you can generally get a small performance improvement by refitting the model on the entire training set.

Conveniently, GridSearchCV from sklearn will automatically refit the model with the best set of hyperparameters using the entire training set.

This functionality is ON by default, but you can confirm it:

Confirm model will be retrained

Python

1

2

printclf.refit

# True

Now, you can simply use the
clf object as your model when applying it to other sets of data. That's what we'll be doing in the next step.

Step 9: Evaluate model pipeline on test data.

Alright, we're in the home stretch!

This step is really straightforward once you understand that the
clf object you used to tune the hyperparameters can also be used directly like a model object.

Here's how to predict a new set of data:

Predict a new set of data

Python

1

y_pred=clf.predict(X_test)

Now we can use the metrics we imported earlier to evaluate our model performance.

Python

1

2

3

4

5

printr2_score(y_test,y_pred)

# 0.45044082571584243

printmean_squared_error(y_test,y_pred)

# 0.35461593750000003

Great, so now the question is... is this performance good enough?

Well, the rule of thumb is that your very first model probably won't be the best possible model. However, we recommend a combination of three strategies to decide if you're satisfied with your model performance.

Start with the goal of the model. If the model is tied to a business problem, have you successfully solved the problem?

Look in academic literature to get a sense of the current performance benchmarks for specific types of data.

Try to find low-hanging fruit in terms of ways to improve your model.

There are various ways to improve a model. We'll have more guides that go into detail about how to improve model performance, but here are a few quick things to try:

Engineer smarter features after spending more time on exploratory analysis.

Speak to a domain expert to get more context (...this is a good excuse to go wine tasting!).

As a final note, when you try other families of models, we recommend using the same training and test set as you used to fit the random forest model. That's the best way to get a true apples-to-apples comparison between your models.

Step 10: Save model for future use.

Great job completing this tutorial!

You've done the hard part, and deserve another glass of wine. Maybe this time you can use your shiny new predictive model to select the bottle.

But before you go, let's save your hard work so you can use the model in the future. It's really easy to do so:

Save model to a .pkl file

Python

1

joblib.dump(clf,'rf_regressor.pkl')

And that's it. When you want to load the model again, simply use this function:

Load model from .pkl file

Python

1

2

3

4

clf2=joblib.load('rf_regressor.pkl')

# Predict data set using loaded model

clf2.predict(X_test)

Congratulations, you've reached the end of this tutorial!

We've just completed a whirlwind tour of Scikit-Learn's core functionality, but we've only really scratched the surface. Hopefully you've gained some guideposts to further explore all that sklearn has to offer.

Enjoyed the tutorial?

If you enjoy this type of hands-on learning, then you'll love our new Machine Learning Masterclass. It's sorta like this, except more epic, more comprehensive, more detailed, more interactive, and packed with much more information.