In grsec version 4.7+ I cannot get any desktop environment loading successfully. It worked after disabling ioperm restrictions so I guess its a conflict with Xorg. This change is not needed with 4.5 hardened series.