scikit-learn video #9: Better evaluation of classification models

Welcome back to my video series on machine learning in Python with scikit-learn. In the previous video, we learned how to search for the optimal tuning parameters for a model using both GridSearchCV and RandomizedSearchCV.

In this video, you'll learn how to properly evaluate a classification model using a variety of common tools and metrics, as well as how to adjust the performance of a classifier to best match your business objectives. Here's the agenda:

Video #9: How to evaluate a classifier in scikit-learn

What is the purpose of model evaluation, and what are some common evaluation procedures?

What is the usage of classification accuracy, and what are its limitations?

How does a confusion matrix describe the performance of a classifier?

What metrics can be computed from a confusion matrix?

How can you adjust classifier performance by changing the classification threshold?

What is the purpose of an ROC curve?

How does Area Under the Curve (AUC) differ from classification accuracy?

Remember that the goal of model evaluation is to help you estimate how well a particular model will generalize to new data so that you can choose between different models. (Watch video #5 for a refresher.) Not only do you need an evaluation procedure such as train/test split or cross-validation, but you also need an evaluation metric in order to quantify model performance.

One example of a classification metric is accuracy, which is simply the percentage of correct predictions. Despite its popularity, classification accuracy obscures two critical pieces of information: the underlying distribution of your response values, and the "types" of errors your classifier is making. As such, classification accuracy does not give you a clear picture of how your classifier is actually performing.

A solution to this problem is the confusion matrix, which is essentially a tally of the types of correct and incorrect predictions that the classifier has made:

Although the confusion matrix is not itself as evaluation metric, there are many possible evaluation metrics that can be calculated from a confusion matrix: sensitivity (or "recall"), specificity, precision, and others, all of which are covered in the video. The key point here is that a confusion matrix enables you to choose the particular evaluation metric that best matches the goals of your project.

To understand your classifier at an even deeper level, you need to calculate "predicted probabilities of class membership", which are essentially your classifier's judgment of the likelihood that each testing observation belongs to each class:

Calculating predicted probabilities actually enables you to adjust the performance of your model simply by changing the "threshold value" at which a zero or a one are predicted! This is a simple but important technique that is often overlooked during the machine learning process.

The final evaluation tool presented in the video is the ROC curve, which is frequently misunderstood despite its broad usage. It is actually just a simple extension of the "thresholding" idea above, in which sensitivity is plotted against 1 minus specificity for all possible classification thresholds:

It turns out that the percentage of the plot that is underneath the ROC curve, also known as "Area Under the Curve" (AUC), is a useful alternative to classification accuracy as a single number summary of classifier performance.

Classifier evaluation is a substantial field, but understanding the basic tools and metrics above will give you a solid foundation for intelligently evaluating your classification model. Check out the many resources below if you'd like to go deeper into this material, and let me know in the comments if you have any questions!

Conclusion of the series

At least for the time being, this is actually going to be my final video in the scikit-learn series. Thank you so much for joining me throughout the series, and thank you also for all of your kind comments!

However, I'm at this last video and I'm struggling to understand the custom function you created.

"def evaluate_threshold(threshold):

print('Sensitivity:', tpr[thresholds > threshold][-1])

print('Specificity:', 1 - fpr[thresholds > threshold][-1])"

I do not understand tpr[thresholds > threshold][-1]

Ritchie Ng

I went through Andrew Ng's course. It gives a very good theoretical foundation so you can move on to learn the tools that seem like blackboxes to most but you would understand how they work in theory. You should start with his course. Even though it's in Octave, you'll realize that he chose this because it's the easiest to start with without imports such as not worrying what to import.