IRIX Binary Compatibility, Part 2

Unix Program Startup

Now that our kernel is able to distinguish the difference between IRIX
binaries and other programs, we need to arrange the program environment so that the IRIX binary is able to start up. (See Part 1 in this series for more on this.)

Generally speaking, Unix kernels have to communicate a few things to user programs in order to start them up. This includes the program's arguments and
environment, and for dynamic binaries, the ELF auxiliary table, which
is used by the dynamic linker to learn how to link the program. All this
information is transmitted to the user program through the CPU registers
and the stack.

If this information is corrupted, static binaries are still likely to work, but they will lose their arguments and environment. On the other hand, dynamic executables will not start at all if the ELF auxiliary table is screwed, because the dynamic linker will not be able to link them.

Therefore, it is a good idea to start with a simple static binary. We will use
IRIX 5's sed(1) here. When we run it using the NetBSD native function to set up
the stack and CPU register, it is able to start up. Then it gets a SIGSYS
signal and it dumps core on the first system call, because our system call
table for IRIX binaries is still empty. It is possible to check what the
missing system call is with the ktrace(1) command on NetBSD:

Most of the system calls first used by /bin/sed are plain SVR4, so it
was easy to emulate them: just copy the system call definition from
sys/svr4/syscall.master to sys/irix/syscall.master, issue a make to refresh
the files generated from syscall.master, rebuild a kernel, reboot, and retry.

Within a few minutes, it was easy to get IRIX's /bin/sed nearly working. The
next problem was to have it take its arguments correctly.

Setting Up the Stack for Program Startup

For static binaries, the stack is used to transmit arguments and the environment to the user program. The way it should be done is documented in the SVR4 ABI MIPS processor supplement.

In This Series

IRIX Binary Compatibility, Part 6
With IRIX threads emulated, it's time to emulate share groups, a building block of parallel processing. Emmanuel Dreyfus digs deep into his bag of reverse engineering tricks to demonstrate how headers, documentation, a debugger, and a lot of luck are helping NetBSD build a binary compatibility layer for IRIX.

IRIX Binary Compatibility, Part 5
How do you emulate a thread model on an operating system that doesn't support native threads (in user space, anyway)? Emmanuel Dreyfus returns with the fifth article of his series on reverse engineering and kernel programming. This time, he explains thread models and demonstrates how NetBSD emulates IRIX threads.

IRIX Binary Compatibility, Part 1
This article details the IRIX binary compatibility
implementation for the NetBSD operating system. It covers creating a new emulation subsystem inside the NetBSD kernel as well as some reverse engineering to understand and reproduce how IRIX internals work.

The NetBSD kernel uses a function pointed to by the es_copyargs field of
the struct execsw to set up the program stack on startup. Because NetBSD/mips
ports conform to the SVR4 ABI, we could have expected the NetBSD
version of this function (elf32_copyargs() from sys/kern/kern_exec_elf32.c)
to just work with IRIX binaries. Unfortunately, this is not true. Using the
NetBSD elf32_copyargs function with static o32 IRIX binary such as /bin/sed
showed weird behavior with the way argument read: sometimes /bin/sed
was reading the arguments correctly, sometimes it was not. The behavior
was dependent upon the argument length. This suggested that something in
the stack had to be aligned on a particular boundary; with some argument
lengths it was being aligned, and with others it was not.

I already had to face this kind of situation when working on Linux/PowerPC
binary compatibility on NetBSD (read the whole story). However, the situation is different here: IRIX is a closed source proprietary OS; therefore it is not possible to grab kernel sources and look at the way the IRIX kernel sets up the stack. Worse,
because I was not able to build static binaries, it was impossible to make
a static test program that dumped the stack and displayed argc, argv and envp to check what was wrong in the way the stack was set up.

The solution was to use gdb. With gdb we can run /bin/sed on IRIX, set a breakpoint at the beginning of the program and then examine the stack.
The first thing to know is the program startup address. This information
can be obtained using objdump on IRIX's sed:

Then we can start gdb and set our breakpoint. It seems that it is not
possible to break on the program's first instruction, but we can break on
the second instruction. On the MIPS, all instructions are four bytes long,
hence the second instruction is four bytes away from the program's start
address, at 0x100000c0 + 0x4 = 0x100000c4:

In this dump, we recognize a standard startup stack layout--the argc value (three arguments: '/bin/sed', 'aa', and 'aaa'), followed by the argv array (a NULL terminated array of pointers to the argument strings), and then the envp array (a NULL terminated array of pointers to the environment
strings). There are a lot of environment strings here, hence we do not see
the trailing NULL here, it is a bit farther in the stack dump.

Dumping the stack with various arguments to /bin/sed, it is possible to
discover that, for an IRIX binary, the argv[0] must be aligned on a 16-byte boundary. The IRIX kernel sets the stack that way, and IRIX binaries depend
on this particular layout.

It was not possible to modify elf32_copyargs() to implement this particular
behavior because it is also used by native NetBSD binaries. Hence, the
solution was to duplicate what was done in elf32_copyargs() in an
irix_copyargs() function, which can be found in
sys/compat/irix/irix_exec_elf32.c. This irix_copyargs() function just does
elf32_copyargs() job and it enforces the 16-byte alignment of argv[0]. Of
course, the irix_copyargs() function must be used in the es_copyargs field
of the struct execsw for IRIX in sys/kern/exec_conf.c.

With this adjustment, static IRIX binaries were able to read their
arguments and environment correctly.