Logits embedding validation gives NaN loss

labels: Tensor of shape [d_0, d_1, ..., d_{r-1}] (where r is rank of labels and result) and dtype int32 or int64.Each entry in labels must be an index in [0, num_classes).Other values will raise an exception when this op is run on CPU, and return NaN for corresponding loss and gradient rows on GPU.

and I am getting NaNs as loss in GPU and exceptions in CPU mode when using the Logits estimator with embedding_validation=True.
This happens when I run bob tf eval with ReplayMobile. It happens rarely so I don't know what is going on. Here is one error that I get on CPU: