Eager Execution

TensorFlow's eager execution is an imperative programming environment that
evaluates operations immediately, without building graphs: operations return
concrete values instead of constructing a computational graph to run later. This
makes it easy to get started with TensorFlow and debug models, and it
reduces boilerplate as well. To follow along with this guide, run the code
samples below in an interactive python interpreter.

Eager execution is a flexible machine learning platform for research and
experimentation, providing:

An intuitive interface—Structure your code naturally and use Python data
structures. Quickly iterate on small models and small data.

Enabling eager execution changes how TensorFlow operations behave—now they
immediately evaluate and return their values to Python. tf.Tensor objects
reference concrete values instead of symbolic handles to nodes in a computational
graph. Since there isn't a computational graph to build and run later in a
session, it's easy to inspect results using print() or a debugger. Evaluating,
printing, and checking tensor values does not break the flow for computing
gradients.

This has conditionals that depend on tensor values and it prints these values
at runtime.

Build a model

Many machine learning models are represented by composing layers. When
using TensorFlow with eager execution you can either write your own layers or
use a layer provided in the tf.keras.layers package.

While you can use any Python object to represent a layer,
TensorFlow has tf.keras.layers.Layer as a convenient base class. Inherit from
it to implement your own layer:

class MySimpleLayer(tf.keras.layers.Layer):
def __init__(self, output_units):
self.output_units = output_units
def build(self, input):
# The build method gets called the first time your layer is used.
# Creating variables on build() allows you to make their shape depend
# on the input shape and hence remove the need for the user to specify
# full shapes. It is possible to create variables during __init__() if
# you already know their full shapes.
self.kernel = self.add_variable(
"kernel", [input.shape[-1], self.output_units])
def call(self, input):
# Override call() instead of __call__ so we can perform some bookkeeping.
return tf.matmul(input, self.kernel)

Use tf.keras.layers.Dense layer instead of MySimpleLayer above as it has
a superset of its functionality (it can also add a bias).

When composing layers into models you can use tf.keras.Sequential to represent
models which are a linear stack of layers. It is easy to use for basic models:

tf.GradientTape is an opt-in feature to provide maximal performance when
not tracing. Since different operations can occur during each call, all
forward-pass operations get recorded to a "tape". To compute the gradient, play
the tape backwards and then discard. A particular tf.GradientTape can only
compute one gradient; subsequent calls throw a runtime error.

The following example creates a multi-layer model that classifies the standard
MNIST handwritten digits. It
demonstrates the optimizer and layer APIs to build trainable graphs in an eager
execution environment.

Train a model

Even without training, call the model and inspect the output in eager execution:

Use objects for state during eager execution

With graph execution, program state (such as the variables) is stored in global
collections and their lifetime is managed by the tf.Session object. In
contrast, during eager execution the lifetime of state objects is determined by
the lifetime of their corresponding Python object.

Variables are objects

During eager execution, variables persist until the last reference to the object
is removed, and is then deleted.

To save and load models, tfe.Checkpoint stores the internal state of objects,
without requiring hidden variables. To record the state of a model,
an optimizer, and a global step, pass them to a tfe.Checkpoint:

Summaries and TensorBoard

TensorBoard is a visualization tool for
understanding, debugging and optimizing the model training process. It uses
summary events that are written while executing the program.

tf.contrib.summary is compatible with both eager and graph execution
environments. Summary operations, such as tf.contrib.summary.scalar, are
inserted during model construction. For example, to record summaries once every
100 global steps:

Advanced automatic differentiation topics

Dynamic models

tf.GradientTape can also be used in dynamic models. This example for a
backtracking line search
algorithm looks like normal NumPy code, except there are gradients and is
differentiable, despite the complex control flow:

Additional functions to compute gradients

tf.GradientTape is a powerful interface for computing gradients, but there
is another Autograd-style API available for
automatic differentiation. These functions are useful if writing math code with
only tensors and gradient functions, and without tfe.Variables:

tfe.gradients_function —Returns a function that computes the derivatives
of its input function parameter with respect to its arguments. The input
function parameter must return a scalar value. When the returned function is
invoked, it returns a list of tf.Tensor objects: one element for each
argument of the input function. Since anything of interest must be passed as a
function parameter, this becomes unwieldy if there's a dependency on many
trainable parameters.

tfe.value_and_gradients_function —Similar to
tfe.gradients_function, but when the returned function is invoked, it
returns the value from the input function in addition to the list of
derivatives of the input function with respect to its arguments.

In the following example, tfe.gradients_function takes the square
function as an argument and returns a function that computes the partial
derivatives of square with respect to its inputs. To calculate the derivative
of square at 3, grad(3.0) returns 6.

Custom gradients

Custom gradients are an easy way to override gradients in eager and graph
execution. Within the forward function, define the gradient with respect to the
inputs, outputs, or intermediate results. For example, here's an easy way to clip
the norm of the gradients in the backward pass:

Here, the log1pexp function can be analytically simplified with a custom
gradient. The implementation below reuses the value for tf.exp(x) that is
computed during the forward pass—making it more efficient by eliminating
redundant calculations:

Benchmarks

For compute-heavy models, such as
ResNet50
training on a GPU, eager execution performance is comparable to graph execution.
But this gap grows larger for models with less computation and there is work to
be done for optimizing hot code paths for models with lots of small operations.

Work with graphs

While eager execution makes development and debugging more interactive,
TensorFlow graph execution has advantages for distributed training, performance
optimizations, and production deployment. However, writing graph code can feel
different than writing regular Python code and more difficult to debug.

For building and training graph-constructed models, the Python program first
builds a graph representing the computation, then invokes Session.run to send
the graph for execution on the C++-based runtime. This provides:

Most model code works the same during eager and graph execution, but there are
exceptions. (For example, dynamic models using Python control flow to change the
computation based on inputs.)

Once eager execution is enabled with tf.enable_eager_execution, it
cannot be turned off. Start a new Python session to return to graph execution.

It's best to write code for both eager execution and graph execution. This
gives you eager's interactive experimentation and debuggability with the
distributed performance benefits of graph execution.

Write, debug, and iterate in eager execution, then import the model graph for
production deployment. Use tfe.Checkpoint to save and restore model
variables, this allows movement between eager and graph execution environments.
See the examples in:
tensorflow/contrib/eager/python/examples.

Use eager execution in a graph environment

Selectively enable eager execution in a TensorFlow graph environment using
tfe.py_func. This is used when tf.enable_eager_execution() has not
been called.