After I started noticing the "holes" forming, I tried going back and filling them in again. After about 20 or 30 minutes, the holes started forming again. I went back 3 or 4 times, but the holes kept coming back in different areas.

I did not add water to the grout. I mixed it thoroughly before using it. The room was about 65 degrees.