I have installed OpenBSD without the x packages (- x* at packages selection prompt).
Does that mean I have to install X-related stuff too to be able to install packages, that do NOT specifically require X-frontend or GUI?

Another common failure is a missing X11 installation. Even if the port you try to build has no direct dependency on X11, a subpackage of it or its dependencies may require X11 headers and libraries. Building ports on systems without X11 is not supported, so if you insist on doing so, you are on your own to figure it out. For many ports, there are, however, "no_x11" flavored packages available, which you can install without needing X11 on your system.

The more important question is "why are you compiling ports?" Building ports results in the very same packages which are already available on the project mirrors. OpenBSD 5.0-stable installations can utilitize packages from OpenBSD 5.0-release since there are to be no library differences between -release & -stable.