The first of its kind: Harness reinforcement learning in production with Facebook’s Horizon

Have you moved on to reinforcement learning for ML? This new approach to machine learning is now supported by Horizon, an end-to-end platform that has just been open-sourced by Facebook. Let’s broaden our horizons and see what Horizon has to offer!

In this era of big data, sometimes it can be difficult to separate the signal from the noise. Thanks to a new platform open sourced by Facebook, all ML engineers need to do is head for the Horizon.

Horizon is an end-to-end platform for applied reinforcement learning built in Python. It utilizes PyTorch for modeling and training and Caffe2 for model serving. This open source platform includes workflows to train popular reinforcement learning algorithms, along with data preprocessing, feature transformation, distributed training, counterfactual policy evaluation, and optimized serving.

What is reinforcement learning?

If you’ve been following along with our “Know Your History” series, you’ll remember that artificial intelligence has a number of interesting offshoots, including machine learning. Reinforcement learning (RL for short) is the latest attempt to make our algorithms better.

Reinforcement learning’s main focus is on performance. In general, reinforcement learning is an approach based on how software agents should take actions in an environment so as to maximize some notion of cumulative reward.

As the Facebook devs put it, machine learning systems require oversight from engineers to put ML predictions into practice. With reinforcement learning, the system itself can make decisions, take actions, and then adapt those decisions based on feedback.

However, bringing this kind of optimized performance into production has been somewhat difficult. That’s where Facebook’s Horizon comes in.

Horizon’s broad view

Horizon was designed to bridge the gap between the lab and the real world. In particular, Horizon pays close attention to issues that are specific to production environments, making sure they are better supported. This includes features like feature normalization, distributed training, large-scale deployment and serving.

More importantly, Horizon was designed for large, complex data sets and scales wonderfully. Applied RL models are often more sensitive to noisy and unnormalized data than traditional deep networks. Because Horizon preprocesses the data with Apache Spark, developers can use PyTorch based algorithms for normalization and training.

It doesn’t matter how much computing power you have available, either. Horizon works equally well on a single GPU or CPU as it does on large clusters. That said, it is actually optimized for use on large clusters, as distributed training allows developers to solve for more complex problems and iterate the models even faster.