Preventing Executable Files From Compromising Security

A number of security bugs are related to default executable stacks when
their permissions are set to read, write, and execute. While stacks with execute
permissions are allowed, most programs can function correctly without using executable stacks.

The noexec_user_stack variable enables you to specify whether stack mappings are executable. The
variable is available as of the Solaris 2.6 release. By default, this
variable is set to zero, except on 64-bit applications, which provides ABI-compliant
behavior. If the variable is set to a non-zero value, the system marks
the stack of every process in the system as readable and writable,
but not executable.

Once this variable is set, programs that attempt to execute code on
their stack are sent a SIGSEGV signal. This signal usually results in
the program terminating with a core dump. Such programs also generate a
warning message that includes the name of the offending program, the process ID,
and the real UID of the user who ran the program. For
example:

a.out[347] attempt to execute code on stack by uid 555

The message is logged by the syslog daemon when the syslogkern
facility is set to notice level. This logging is set by default
in the syslog.conf file, which means that the message is sent to
both the console and the /var/adm/messages file. For more information, see the
syslogd(1M) and syslog.conf(4) man pages.

The syslog message is useful for observing potential security problems. The message
also identifies valid programs that depend upon executable stacks that have been
prevented from correct operation by setting this variable. If you do not want
any messages logged, then set the noexec_user_stack_log variable to zero in the
/etc/system file. Even though messages are not being logged, the SIGSEGV signal
can continue to cause the executing program to terminate with a core
dump.

You can use the mprotect() function if you want programs to explicitly
mark their stack as executable. For more information, see the mprotect(2) man
page.

Because of hardware limitations, the capability of catching and reporting executable stack
problems is not available on most x86-based systems. Systems in the AMD64
product family can catch and report executable stack problems.