NetBSD has always had portability as a major goal. It's not
surprising that it's able to run binaries compiled for Linux, FreeBSD,
Solaris, and other operating systems. The reasons for this are partly
pragmatic and partly impractical.

Manu graciously agreed to an interview to clarify the project's
goals and the scope of last week's update.

ORN: Let's talk about the basics of binary compatibility
for a moment. Here are the facts as I understand them:

it is intended for programs for which you do not have source code

it almost always requires running on the same architecture

it means emulating a binary interface

most of the work is to write a translator for system and kernel calls
between the two operating systems

For example, you're not going to see Mac OS X apps running on non-PPC
hardware anytime soon, if ever, because the architecture doesn't match.

Is there anything I'm missing?

ED: No, this sounds fine, you are right.

If we want to run a binary on a processor it was not build for, then we have to
emulate the CPU in software. This kind of thing has been done by Apple for
their m68k to PowerPC transition. Emulating the CPU in software costs a lot, so
it will give you reasonable performances only if the native CPU is much faster
than the emulated CPU (which was the case for Apple's m68k emulator for the
PowerPC, but it was an unusual situation).

IRIX Series of Articles

IRIX Binary Compatibility, Part 6
With IRIX threads emulated, it's time to emulate share groups, a building block of parallel processing. Emmanuel Dreyfus digs deep into his bag of reverse engineering tricks to demonstrate how headers, documentation, a debugger, and a lot of luck are helping NetBSD build a binary compatibility layer for IRIX.

IRIX Binary Compatibility, Part 5
How do you emulate a thread model on an operating system that doesn't support native threads (in user space, anyway)? Emmanuel Dreyfus returns with the fifth article of his series on reverse engineering and kernel programming. This time, he explains thread models and demonstrates how NetBSD emulates IRIX threads.

IRIX Binary Compatibility, Part 2
Emmanual Dreyfus shows us how he implemented the things necessary to start an IRIX binary. These things include the program's arguments, environment, and for dynamic binaries, the ELF auxiliary table, which is used by the dynamic linker to learn how to link the program.

IRIX Binary Compatibility, Part 1
This article details the IRIX binary compatibility
implementation for the NetBSD operating system. It covers creating a new emulation subsystem inside the NetBSD kernel as well as some reverse engineering to understand and reproduce how IRIX internals work.

ORN: I know you've been involved with other binary
compatibility layers (most recently your IRIX series, for example). How does
one get started? Are there any applications you really wanted to run on NetBSD
but couldn't?

ED:There are a lot of motivations to write some binary
compatibility software.

The first obvious reason is indeed to get some interesting applications
running on NetBSD. For instance, there is no recent native Java Virtual Machine
(JVM) available for NetBSD PowerPC ports, and we now have a lot of people
developing Java applications because they believe Java to be more portable
than C. I had the need to run Java programs, and the only machine fast enough
to do this at home was my NetBSD/macppc machine. This is what pushed me to
wrote the PowerPC parts of Linux binary compatibility on NetBSD. We are now
able to use the Blackdown Java Developement Kit (JDK) for LinuxPPC on NetBSD
PowerPC ports through the binary compatibility layer.

There is a different motivation to get a binary compatibility layer for
people using UNIX workstations such as Alphas, Suns or SGIs. For various reason
these people might want to migrate from their original Operating Systems (OS):
the box is too old to support recent versions of the OS, or they do not have
the money for an OS upgrade, or the OS support will be dropped in the future,
or they cannot bear the native OS any longer.

NetBSD runs on these workstations. Recent NetBSD releases work well on
old hardware, NetBSD is free, NetBSD is well supported, and NetBSD is
administrator friendly (okay, this latter point is just a matter of
taste). Therefore switching the box to NetBSD can be a good option.
Unfortunately, there are sometimes some old locally developed application
that cannot be migrated easily because they are not written in a portable
way. The binary compatibility layer can help a lot here.

There are also some academic motivations. Implementing Mach binary
compatibility in NetBSD will help us finding out how well a Mach-like
microkernel IPC mechanism can be implemented within a traditional
monolithic kernel such as NetBSD. Christos Zoulas (who set the first stone
on Mach binary compatibility on NetBSD) and I are planning to write a
paper covering the following topics:

performance analysis on mach-ipc implemented on top of a monolithic
kernel;

challenges/complexity of implementing mach ipc messages on a monolithic
kernel.

As you see, I had a lot of motivations. And of course also the
pleasure to work on a challenging project.

ORN:It seems that supporting a "real-world application" on a
fledgling operating system would be a good way to discover what features you
don't yet support. Of course, NetBSD is a mature OS, but have you found any
improvements that needed to be made?

ED:Well, yes and no. When setting up the IRIX binary
compatibility on NetBSD/sgimips, I discovered a bunch of IRIX features that
NetBSD does not support natively. We have been able to support a lot of them,
but there are very few that we really integrated into NetBSD beyond the IRIX
binary compatibility layer.

For instance, IRIX does use a libc-provided signal trampoline (see IRIX
binary compatibility part 4 for more informations about this), and
NetBSD did not. The libc-provided signal trampoline has many advantages,
so we switched to that feature for native NetBSD binaries, thus improving
NetBSD.

On the other hand, there are a lot of foreign OSes feature emulated in the
binary compatibility layer that we do not want to support natively, mostly
because they consume too much resources: the IRIX ability to share all the
Virtual Memory (VM) space mapping but some pages within a share group is an
example of such a feature.

ORN: From my (few) experiences with the Mac-on-Linux project, it looks like the project is similar. Can you speak of the similarities or differences?

ED:Well, MoL is much like the PowerPC equivalent of i386
software such as VMWare or Plex86: these are virtual machines inside which
you run the whole foreign OS, kernel included. It's like another machine inside
your machine.

Binary compatibility makes it possible to run the foreign OS binaries
on top of the NetBSD kernel. It is possible to mix binaries from NetBSD
and the emulated OSes: they are able to communicate through pipes or
sockets, you see them in the process list with ps, a foreign OS
binary is able to launch a native process, and so on. Binary compatibility
enable me to run an anti-virus software build for Linux as a filter in my
NetBSD mail server, for instance.

With virtualization software such as MoL or VMWare, the guest
Operating System in the virtual machine is completely separated from the
host OS, there is no way to bring the processes from the two OSes into
interaction. This has some advantages, if you want to study some hostile
software, but on the other hand you will never have the integration you
have with binary compatibility.

Of course this integration is theorical when we come to OSes such as
MacOS 9, which have nothing common with UNIX: it makes no sense to connect
a MacOS application to a UNIX application through pipes for example. So
virtualization is probably the best way to run MacOS 9 binaries on a UNIX
system. For MacOS X, which is UNIX-based, I believe binary compatibility
would be much more interesting than virtualization.

There is another point to comment on: performance. Binary
compatibility is extremely efficient. The performance loss compared to
running the binary on the OS it was build for is about 1-2%. And for some
system calls, the NetBSD compatibility implementation is even faster than
the original implementation.

Virtualization performance is good, but not as good as binary
compatibility performance. Virtualization software has to catch all
hardware exceptions generated by the guess OS and emulate them in
software. In some situations, it can be quite expensive.

ORN:Even though you don't have the source code of
applications or libraries you'd like to run on your system, if they don't
come directly from the OS vendor, there has to be documentation somewhere.
How much can you learn just by reading public docs on kernel interfaces?
Does this help?

ED:A lot! In fact, all the necessary information is
found in the kernel header files and the man pages. Here is how it
happens: you try running a given binary, and you hit some system call you
did not implement yet. All that you have is the system call number and
the arguments the binary used when issuing the system call.

In the kernel header files, you find the system call table. From the
system call number, you get the system call name. If you are lucky, the
system call has a man page, which refers to some kernel header file for
the data structures the system call uses. You have all the documentation
about the system call, and you just implement it based on its
documentation. Your life is easy.

If you are not lucky, the system call is undocumented. Then you have
to study it using gdb, trying to give it different arguments,
and checking for the results. Usually you can figure out a bit about it
based on its name; MacOS X's pthread_exit() must have
something to do with thread termination, for example. And sometimes the
system call is not documented, but you can find its prototype and the
data structures it uses in a kernel header. That helps a lot.

Contrary to popular belief, having the kernel sources for the OS you
want to emulate does not help very much. I experienced that with Linux.
Sometimes you have a behavior which is not documented. You can read messy
Linux sources and try to figure out how this is supposed to behave, or you
can run some test programs on a live system and inspect things with
gdb. The latter tends to produce much more interesting results, since you
actually see how things are, instead of trying to understand how they
should be.

ORN:What have been the responses so far? Have you
helped people migrate to NetBSD without losing their necessary
applications?

ED:I had a lot of feedbacks about the LinuxPPC binary
compatibility, mostly from people running Java applications such as Cocoon
or FOP.

IRIX binary compatibility is more recent, and less mature. There are
the N32 binaries that are not supported yet which makes it rather
impractical for replacing IRIX with NetBSD. But I know some people who
think about it.

ORN: Conversely, have you had anyone (IRIX or HP engineers)
contact you to offer help?

ED: Some SGI engineers contacted me to wish me some
success on the IRIX binary compatibility project, but I got no help from
them on the technical front. I had no news from Apple engineers yet.

ED: Well, as far as I know (I am not a lawyer) reverse
engineering some software for interoperability purpose is a legal practice
in a lot of countries around the world, including my home country. There
is indeed some concern in the U.S. because of the DMCA.

IRIX binary compatibility is a descendant of System V Release 4 (SVR4)
binary compatibility, which was developed pre-DMCA, so there should be few
problems. For MacOS X binary compatibility, things are different: most of
the MacOS X kernel is from the Carnegie Mellon University (CMU) Mach
project and is available under the CMU copyright. There is very little
Apple proprietary information involved, so it should be all right too.

But, generally speaking, there has been no legal threat against
emulation projects, whether commercial or noncommercial. If that happens,
one could argue that it is some kind of consumer right to run on any
platform some software for which one purchased a license. A bit like you
expect to play a CD you purchased on any CD-Player, whether it is from
Pioneer or from Sony.

ORN:How portable are your compatibility layers? On
the one side, any Mac OS X library is going to target the same Darwin/Mach
system calls. On the other side, you'll need specific code for, say, the
NetBSD and the Linux kernels. Given that the binary interface is hardest
to find, how difficult would it be to port the code to another free Unix?
(I'm assuming that figuring out what you need to implement is the hardest
part. If not, please correct me.)

ED:You are mostly right, that is the hardest part and
also the most interesting. Porting to another UNIX system would be an
extremely boring job: nothing to discover, just thousands lines of code to
tweak.

There would be a lot of tweaking because the binary compatibility
layer is indeed not portable. And it is not portable because it is
implemented in the kernel. There are some standards that help make
userland programs portable, but there is nothing which standardizes
internal kernel APIs between different UNIX systems.

ORN:Anything else we should know?

ED:Yes. NetBSD binary compatibility will enable running MacOS X binaries on
NetBSD. You will still need a MacOS X license. The binary compatibility
only emulates the kernel's behavior, not the userland. Therefore, in
order to run a dynamically-linked application, you need the libraries
from the emulated OS. The libraries are part of the emulated OS, and if
you use them, you need a license for it.

I also need some manpower to help on this
project. Mach emulation is much different from most of the binary
compatibility we already have in NetBSD. We used to emulate other UNIX
kernels, and Mach is not a UNIX kernel. (Darwin, which runs on top of
Mach, is a UNIX kernel.) This makes the project a real challenge.

There are a lot of ways to help, including tasks that are limited to
userland programming. A test suite of Mach traps would be wonderful, for
example. So do not be shy, join the project!