How to Use Metrics for Deep Learning with Keras in Python

The Keras library provides a way to calculate and report on a suite of standard metrics when training deep learning models.

In addition to offering standard metrics for classification and regression problems, Keras also allows you to define and report on your own custom metrics when training deep learning models. This is particularly useful if you want to keep track of a performance measure that better captures the skill of your model during training.

In this tutorial, you will discover how to use the built-in metrics and how to define and use your own metrics when training deep learning models in Keras.

After completing this tutorial, you will know:

How Keras metrics work and how you can use them when training your models.

How to use regression and classification metrics in Keras with worked examples.

How to define and use your own custom metric in Keras with a worked example.

Let’s get started.

Metrics and How to Use Custom Metrics for Deep Learning with Keras in PythonPhoto by Indi Samarajiva, some rights reserved.

Tutorial Overview

This tutorial is divided into 4 parts; they are:

Keras Metrics

Keras Regression Metrics

Keras Classification Metrics

Custom Metrics in Keras

Keras Metrics

Keras allows you to list the metrics to monitor during the training of your model.

You can do this by specifying the “metrics” argument and providing a list of function names (or function name aliases) to the compile() function on your model.

For example:

1

model.compile(...,metrics=['mse'])

The specific metrics that you list can be the names of Keras functions (like mean_squared_error) or string aliases for those functions (like ‘mse‘).

Metric values are recorded at the end of each epoch on the training dataset. If a validation dataset is also provided, then the metric recorded is also calculated for the validation dataset.

All metrics are reported in verbose output and in the history object returned from calling the fit() function. In both cases, the name of the metric function is used as the key for the metric values. In the case of metrics for the validation dataset, the “val_” prefix is added to the key.

Both loss functions and explicitly defined Keras metrics can be used as training metrics.

Keras Regression Metrics

Below is a list of the metrics that you can use in Keras on regression problems.

Running the example reports the custom RMSE metric at the end of each training epoch.

1

2

3

4

5

6

7

8

9

10

11

...

Epoch 496/500

0s - loss: 1.2992e-06 - rmse: 9.7909e-04

Epoch 497/500

0s - loss: 1.2681e-06 - rmse: 9.6731e-04

Epoch 498/500

0s - loss: 1.2377e-06 - rmse: 9.5562e-04

Epoch 499/500

0s - loss: 1.2079e-06 - rmse: 9.4403e-04

Epoch 500/500

0s - loss: 1.1788e-06 - rmse: 9.3261e-04

At the end of the run, a line plot of the custom RMSE metric is created.

Line Plot of Custom RMSE Keras Metric for Regression

Your custom metric function must operate on Keras internal data structures that may be different depending on the backend used (e.g. tensorflow.python.framework.ops.Tensor when using tensorflow) rather than the raw yhat and y values directly.

For this reason, I would recommend using the backend math functions wherever possible for consistency and execution speed.

Further Reading

This section provides more resources on the topic if you are looking go deeper.