Static vs. Dynamic Inference

You can choose either of the following inference strategies:

offline inference, meaning that you make all possible predictions
in a batch, using a MapReduce or something similar. You then
write the predictions to an SSTable or Bigtable, and then feed these
to a cache/lookup table.

online inference, meaning that you predict on demand, using a server.

Estimated Time: 3 minutesLearning Objectives

Understand the pros and cons of static and dynamic inference.

Estimate training and serving needs for real-world scenarios.

Learn more about static vs. dynamic inference in the following video (2 min).

Static vs. Dynamic Inference

ML System Paradigms: Inference

Offline Inference

Make all possible predictions in a batch, using a mapreduce or similar.

Write to a table, then feed these to a cache/lookup table.

Online Inference

Predict on demand, using a server.

ML System Paradigms: Inference

Offline Inference

Make all possible predictions in a batch, using a mapreduce or similar.

Write to a table, then feed these to a cache/lookup table.

Upside: don't need to worry much about cost of inference.

Upside: can likely use batch quota.

Upside: can do post-verification of predictions on data before pushing.

ML System Paradigms: Inference

Offline Inference

Make all possible predictions in a batch, using a mapreduce or similar.

Write to a table, then feed these to a cache/lookup table.

Upside: don't need to worry much about cost of inference.

Upside: can likely use batch quota.

Upside: can do post-verification on predictions on data before pushing.

Downside: can only predict things we know about -- bad for long tail.

Downside: update latency likely measured in hours or days.

ML System Paradigms: Inference

Online Inference

Predict on demand, using a server.

Upside: can predict any new item as it comes in -- great for long tail.

ML System Paradigms: Inference

Online Inference

Predict on demand, using a server.

Upside: can predict any new item as it comes in -- great for long tail.