What Is Generative Machine Learning?

Deep Learning has driven the AI/Machine Learning revolution because it works well. But it only works when powered by enough data.

...the rocket engine is the deep learning models and the fuel is the huge amounts of data we can feed to these algorithms.

— Andrew Ng

As Andrew Ng, Founder of Coursera, Google Brain and former Head of AI at Baidu, has said:

"I think AI is akin to building a rocket ship. You need a huge engine and a lot of fuel. If you have a large engine and a tiny amount of fuel, you won’t make it to orbit. If you have a tiny engine and a ton of fuel, you can’t even lift off. To build a rocket you need a huge engine and a lot of fuel.

The analogy to deep learning is that the rocket engine is the deep learning models and the fuel is the huge amounts of data we can feed to these algorithms."

Do you want to use deep learning but are worried about how much labeled training data you'll need to make it work? If you want to use your data to derive business value, getting enough of it labeled to make Deep Learning an option can be prohibitively expensive.

The simple truth is that in order for deep learning to work well, it needs large amounts of labeled data.

Train discriminative models with less data.

In the real world, often only a small portion of your data is labeled. Discriminative deep learning models only work on labeled data (red and green). Can you see the structure here?

Have you wondered why large datasets are so important? It’s because most current deep learning algorithms are discriminative and you need to train your model with large amounts of labeled data to prevent overfitting. Only then can you apply your model outside your training set and obtain reliable novel insights and business value.

Only then can you apply your model outside your training set and get novel insights and business value… which is why you were interested in AI and machine learning in the first place.

However, by building models of how data is generated and how labels are assigned, we can train accurate discriminative models with far less data.

The figures illustrate a simple example. Labels here are indicated by color (red or green). Even with many examples, the rule which discriminates between the labels (color) is not clear. However, if we provide unlabeled data (in white) the labeling rule is clear - one spiral is labeled red and the other green. In fact, with access to the unlabeled data only two labeled examples (to label each spiral) are sufficient to build an accurate model. Generative models uncover this structure, allowing for reliable prediction with far less data.

This is the power of generative learning.

Does this help? Generative learning models ingest unlabeled data (white) and output structure, making predicting class labels much easier without having to label all your samples.

Not only can you use generative machine learning to build accurate discriminative models, but generative models also discover useful insights into the structure and categories hidden within your data.

It is precisely these insights that allow for training with less data.

In many application areas the goal is not simply to predict specific outcomes, but to understand the underlying structure. Models which uncover structure may be much more valuable than predictions alone.