Auxiliary states are special states of symbols that do not correspond to an argument,
and are not updated by gradient descent. Common examples of auxiliary states
include the moving_mean and moving_variance in BatchNorm.
Most operators do not have auxiliary states.