IRIX Binary Compatibility, Part 3

IRIX Oddities: system calls that you will not see anywhere else!

Now that we are able to launch dynamic binaries, the goal is to get
them linking. The dynamic linker has to do a lot of system calls before actually launching the program. Most of them are plain SVR4, and hence are taken from sys/compat/svr4. Here, we will deal with IRIX-specific system calls.

syssgi() Overview

One of the very first things on which we fail when running IRIX 6.5
binaries is the syssgi(2) system call. In fact, syssgi(2) is more like a meta-system call. Its first argument is an int named request. Depending on request's value, syssgi(2) will run literally dozens of different commands. The remaining arguments to syssgi(2) are interpreted according to the request argument.

syssgi(2) commands include some quite standard functionality that is
implemented in plain system calls on NetBSD, such as getgroups(2),
getsid(2), or getpid(2). There are also some SGI-specific things, such as commands to get hardware inventory, system configuration, or NVRAM values.

In This Series

IRIX Binary Compatibility, Part 6
With IRIX threads emulated, it's time to emulate share groups, a building block of parallel processing. Emmanuel Dreyfus digs deep into his bag of reverse engineering tricks to demonstrate how headers, documentation, a debugger, and a lot of luck are helping NetBSD build a binary compatibility layer for IRIX.

IRIX Binary Compatibility, Part 5
How do you emulate a thread model on an operating system that doesn't support native threads (in user space, anyway)? Emmanuel Dreyfus returns with the fifth article of his series on reverse engineering and kernel programming. This time, he explains thread models and demonstrates how NetBSD emulates IRIX threads.

IRIX Binary Compatibility, Part 2
Emmanual Dreyfus shows us how he implemented the things necessary to start an IRIX binary. These things include the program's arguments, environment, and for dynamic binaries, the ELF auxiliary table, which is used by the dynamic linker to learn how to link the program.

IRIX Binary Compatibility, Part 1
This article details the IRIX binary compatibility
implementation for the NetBSD operating system. It covers creating a new emulation subsystem inside the NetBSD kernel as well as some reverse engineering to understand and reproduce how IRIX internals work.

The big question is why SGI decided to fold so much functionality into a single system call. There must be a good reason for doing so, but it is not easy to guess. The only thing that is obvious when you are doing some reverse engineering on syssgi(2) is that for many requests, you do not know what arguments are used when calling syssgi(2). The only information available is the name of the request, and it makes things much more difficult.

syssgi(2) emulation in NetBSD is done in sys/compat/irix/irix_syssgi.c:irix_sys_syssgi().
It is just a giant switch on the request value, which will branch to
various kernel functions implementing the request. All standard features, such as getpid(2), are quite easy to implement. Others are more tricky.

The first difficulty with syssgi(2) is the ELFMAP request. The dynamic
linker invokes syssgi(2) with this request, and all we can find in the
syssgi(2) man page is that this is an interface to implement a system
library function, and that this interface is subject to change. Not
very helpful.

Fortunately, Linux already tried to get there, and the person that
worked on it managed to discover that ELFMAP takes a file descriptor, an ELF program header array, and the array length, and then maps the ELF sections described in the array in the calling process' user space. Information on this can be found inside Linux kernel sources, in
linux/arch/mips/kernel/sysirix.c:irix_syssgi() and
linux/arch/mips/kernel/irixelf.c:irix_mapelf().

In fact, syssgi(ELFMAP) is a kernel implementation of a part of the
dynamic linker. Native binaries on a NetBSD system map each code section doing a mmap(2). Here again, one could wonder what the reasons are for pushing that code from userland to the kernel. One reason could be to improve performance by saving system calls: an IRIX binary can map a library with only one system call.

Reverse Engineering syssgi(ELFMAP)

Another way of guessing what the syssgi(ELFMAP) function does is to use the par(1) command in IRIX. This command is similar to ktrace(1) on NetBSD: it reports the system call activity of a user program. Fortunately, syssgi(ELFMAP) gets disassembled into a more system-call-looking presentation: