If you want to disable it without rebooting, then (still doing as Dude recommended) also comment out the line that loads X. E.g.

# Run xdm in runlevel 5
# x:5:respawn:/etc/X11/prefdm -nodaemon

Then use the telinit q command to have the kernel reread the +/etc/inittab+ file. Any respawn daemons that no longer applies are automatically terminated - which means xdm will be shut down.

Keep in mind to uncomment the line after the 1st reboot in order for runlevel 5 to support X. Personally, I do not on servers as I refuse to have sysadmin run GUIs on servers in a cabinet in the computer room somewhere. If X is needed, they need to use VNC or some other remote frame buffer app that supports virtual X desktops.