Indeed, STATE_LOGGING_VIDEO_MP4 is only available when using OpenGL (GUI or SHARED_MEMORY).
Thinking of it, we could implement this also in DIRECT mode, using the TinyRenderer to generate images.

Some people log the state of objects, and create the video/playback afterwards.

Another option is to use PyBullet.getCameraImage (which also works in DIRECT mode, it uses a CPU renderer) and write those images to create a video using Python. You can write individual images to disk and use a tool to convert it into a video. OpenAI Gym has also some utility for this, various PyBullet Gym environments provide the 'render' method.