"no_x11" packages do not require X, but building the packages from ports typically does.

If you must build this on the server, rather than on a separate build machine, you must install the X file sets to do so. After your no_x11 packages have been built/installed, you may remove X manually if desired.