Subscribe to this blog

Follow by Email

ERNOS: Erlang Networked Operating System

I've been reading Dreaming in Code lately, and I really like it. If you're not a dreamer, you may safely skip the rest of this post ;)

In Chapter 10, "Engineers and Artists", Alan Kay, John Backus, and Jaron Lanier really got me thinking. I've also been thinking a lot about Minix 3, Erlang, and the original Lisp machine. The ideas are beginning to synthesize into something cohesive--more than just the sum of their parts.

Now, I'm sure that many of these ideas have already been envisioned within Tunes.org, LLVM, Microsoft's Singularity project, or in some other place that I haven't managed to discover or fully read, but I'm going to blog them anyway.

Rather than wax philosophical, let me just dump out some ideas:

Start with Minix 3. It's a new microkernel, and it's meant for real use, unlike the original Minix. "This new OS is extremely small, with the part that runs in kernel mode under 4000 lines of executable code." I bet it's written well enough to provide the basis for some fun hacking.

Run Erlang as a daemon in userland. Erlang has the sort of "ridiculous number of 9s reliability" that I'm looking for.

Step by step, move everything into Erlang ;)

Hence, almost everything is interpreted. This is also inspired by Java, .NET, and LLVM. Unlike Java and .NET, the interpreter uses high level P-codes, more like Python.

Because of this, hardware-based "protected-mode" isn't nearly so important. I'm still debating whether it's even necessary to have more than one context (i.e. hardware-based address space) in userland since the Erlang interpreter provides this on another level.

Processes are like Erlang "processes": they're dirt cheap, they don't share a common namespace, and they have really nice message passing semantics. After all, if we're going to be dealing with 100s of CPU cores in a few years, it makes sense to get away from the hard way of doing threads (mutexes in C) and embrace the easy way of doing "threads" (a la Erlang).

Erlang processes have no shared namespace, but they can easily pass data amongst themselves--not just streams of bytes, but full-on data structures. Imagine taking advantage of this between every "program" on your system.

Due to the nature of Erlang data, passing data between programs is copy on write.

Like a Palm, every "program" is running at all times. Unlike a Palm, things are running in an interpreter, so you don't need cooperative multitasking. Nor will a single program bring down the whole system. Rather, the interpreter itself can "context switch" after every few instructions. Since the "processes" are super light-weight, context switching is very fast and happens a lot more frequently.

Since every program is running at all times, that means every program is implicitly started when booting. It's more like turning on a Palm pilot than booting a computer.

Throw away the filesystem. Every "program" has every "document" implicitly loaded at all times. The documents don't have to be serialized to a stream of bytes. They're always meaningfully structured like Lisp data.

Unlike object-oriented systems today that have many objects and few threads, unify the concepts of objects, threads, and modules into something I call a "cell".

That means every object has its own thread.

"Programs" are constructed from a collection of such objects.

Since every object is like a module, it can be reloaded independently of everything else. This has been done in both Erlang and in the Lisp machines.

Since every program is always running and is composed of objects that can be independently reloaded, the objects behave more like biological "cells". Sometimes you get new cells. Sometimes old ones die. However, in general, things are pretty "static". Failures are treated more like a cancer than a stroke.

These "cells" know how to do continuations that can be journaled to disk. That means that if the system needs to restart, the cells restart at their last "checkpoint".

Naturally, the message passing in Erlang works seamlessly between machines. Hence, the "N" in "ERNOS".

Programs that create "documents" for view by other people on other systems do so in an applet-like manner. That is, the code "goes with" the data (a la object oriented programming). The receiver doesn't need to have the same program installed. However, I will assume that non-ERNOS users can install Erlang in the same way most people have a JVM and a Flash player installed. Of course, the normal sandboxing applies.

Comments

Hi,I'm rather fond of OS design so I thought I'd chime in with a few things.

First, what about interacting with devices? In parallel with *nix's "everything's a file" philosophy, would you make mounting a device create another thread on the background to handle all communications with the device?

What about security? What are you envisioning as a way to offer protections? I'm still pretty ignorant of Erlang, so I don't know if there's anything it offers in terms of dispatching over messages by sender that can offer a primitive ACL kind of security.

Also, if I understood you correctly then every program in ERNOS is going to share the same virtual address space?

Your ideas are very different than mine, so it takes a little bit to wrap my head around them. I've been thinking a bit about a capability based OS based on Haskell's type system and possibly using arrows as a way to organize the system.

If you feel like bouncing ideas off of a random dude on the internet, please feel free to e-mail me. I believe it's listed in my profile.

This is going to be a PITA, but is not impossible. The best bet it to apply capability-semantics to Erlang. It is already part of the way there, but would require a bit of heavy lifting to get the various libraries like OTP converted to use capability-secure pids. There was a discussion on erlang-questions a while back in which Mark Miller outlined the short list of changes that would be necessary.

If you considered going in this direction then a lot of the experiences of the KeyKOS and EROS projects would be worth examining.

In Erlang "everything's a process", basically. Whether that's implemented as a thread or a process or something in the VM is dependent on what exactly you're talking to and how the port (driver) was written.

Erlang doesn't provide any security against malicious code within the VM really. Any process can send messages to or (attempt to) kill any other process if it can somehow acquire a reference to that process. Acquiring such references isn't exactly easy, but they're predictable in current versions of the VM.

You could pretty easily implement security at the node level when you're communicating across VMs by simply sending serialized terms over TCP or something rather than using the baked in distribution mechanism (which implies trust via shared secret cookie). The built-in distribution allows arbitrary inter-process messaging, spawning, killing, etc.

"Security" in Erlang is typically done at the compiler level (e.g. epp_dodger) not at the runtime. There was a proposal for a capability based Erlang at some point somewhere, but I don't think anything really came of it. There are things you could do to implement ACLs and such (message-checking proxies maybe), but you'd eventually have to do something about the fact that any process can kill any other process, and that process ids are relatively guessable (sequential).

All in all, it's a very "consenting adults" runtime with regard to security. It's really not usually so much of a consideration because most Erlang environments aren't accepting arbitrary code and the only distributed communication they're doing (using the baked-in mechanisms) is in a very trusted LAN environment, or even amongst different nodes on the same machine. Internet communication is most commonly done with internet protocols, which are data only...

> I'm rather fond of OS design so I thought I'd chime in with a few things.

Oh, good! Someone to reveal how little I know! ;)

> First, what about interacting with devices? In parallel with *nix's "everything's a file" philosophy...

Instead of everything's a file, everything's an object. Like Minix 3, there's a user space program for every driver. Most user code doesn't talk to drivers directly. Writing Erlang code, you'd use Erlang libraries like usual. Those libraries know how to talk to the kernel in the appropriate way, which in turn talks to the user space drivers. Yes, mounting a device does result in another userland program.

> What about security?

My approach would be similar to the Java approach. With interpreted code, it's easier to apply ACLs and put things in sandboxes.

> Also, if I understood you correctly then every program in ERNOS is going to share the same virtual address space?

At a hardware level, yes. Erlang itself entirely separates the processes from one another. Remember, since everything is interpreted, it's safe to trust the interpreter to enforce this separation.

> Your ideas are very different than mine, so it takes a little bit to wrap my head around them.

I applaud your open mindedness. You haven't even called me an idiot yet! ;)

> I've been thinking a bit about a capability based OS based on Haskell's type system and possibly using arrows as a way to organize the system.

Don't be surprised if I admit I don't know what you mean. I know Haskell, a bit, but I haven't used arrows. I don't know what a capability based OS is.

> If you feel like bouncing ideas off of a random dude on the internet...

Can I add you to my IM client? I'm on YIM, AIM, jabber.org, GMail.com, etc. all under "jjinux".

For the capability semantics, check out the erlang-questions archive starting in June 2006 (http://www.erlang.org/ml-archive/erlang-questions/200606/msg00092.html) and hit the erights list at approximately the same time. MarkM didn't quite write the book on capability security (just the Ph.D. thesis, think of his as the Joe Armstrong of capability security...) but he definitely knows what he is talking about. You should also take a look at SSerl for an early approach to fixing Erlang security problems.

Personally, I think that a "secure" erlang would be easier to pull off and would be necessary for any sort of erlang OS. I also happen to know that the bay area capability researchers are aware of and interested in erlang, so putting such a project together might not be a huge task on the same order as cretaing an OS but you would get a lot more help on the former than the latter :)

Regarding the security issues, what would probably need to happen at some level is different "groups" of processes would need to run on different nodes on the same physical device, rather than everything being on the same node. Inferno uses an ssl-based encryption system for Styx/9P2000 (the protocol) -- anyone attempting this with Erlang would be studying Inferno heavily anyway.

Also, rather than get rid of the OS for something hobby-like, you would take advantage of it for the hardware abstraction (you don't want to bother with device driver hell when Linux has done so much of that work for you -- the importance of this can't easily be understated. Plan 9's been around for nearly 2 decades, and it still has crap in the way of driver support).

I agree, I don't *want* to be in device driver hell, but I'd prefer if all the device drivers were userland applications written in Erlang that simply had libraries for doing things like writing to a port or a mem mapped device.

Everyone says that strings are slow in Erlang but I haven't seen anyone actually prove it. I use Erlang, I do a bit of string manipulation using the strings-as-lists paradigm, and it's plenty fast. I know that binaries can be faster, especially if you don't change the string, and that's what iolists are for. Just because binaries can be faster doesn't mean strings are slow, though.

Unicode and ByteString plus the idea of manipulation are kinda mutally exclusive. Are you talking about UTF-8 or something? Or not bytes? I mean, lists-as-string is basically a UCS-4 unicode string implementation. Sure, there's a bit more overhead with a (linked) list versus an array, but a sufficiently advanced VM (like Erlang+HiPE) probably does a very good job at mitigating that.

In other words, if you think Erlang NEEDS some other way to represent strings... prove it. I don't believe you. If "faster strings" was as necessary as you think they are, someone would've implemented them some time in the past *20 years* and got them into Erlang/OTP.

hi, I had a few ideas that will probably take a long time to implement so if you're going in this direction, I'd certainly be happy to share with you.(give to you)I was thinking of an OS with a very small kernel, something like a scheduler and few necessities in order to make it reliable, but hearing of minix3 it would be wise to be based off it.the applications however will not be able to write to disk.the apps will be simple scripts in whatever language, but they will talk using some protocol, perhaps rpc, to the system servers. the system servers provide all the functionality that is needed. there will be an api server as well as media server network server etc. a bit similar to how BeOS used to do the userland servers. servers need to be reliable so they'll be written in erlang. I also wanted this. since everything goes through the servers, applications only have the available commands from the api to work with - so they shouldn't do any damage. this can be enforced with capability based rules for every application, perhaps updated from a central server. (maybe checksummed to make sure the script is the same)updating in general could use the erlang style of code swap where after update the server will use the new code for every incoming request.in addition, I think there could be some database - a bit like cvs maybe - but for the operations you've done until now. so if something crashes, (or u just close the computer) the system is able to open the right applications, with the right documents that were opened, perhaps even at the right cursor point. but this is not saved to memory. simply written to the database after every operation you make.what else?? oh yeah, gui - I really liked the simplicity of the Clean language UI programming, but perhaps it is time to simply make them local web applications using something like the web server yaws and something on the local side to implement all the drawing and effects (web apps usually use js for that, but there is no reason it can't be done in another more suitable language)my original vision had workspaces that you move to using the F keys, and another one that has the application components. users can easily connect these components and provide useful scripts.the original workspaces can be updated remotely, and they'll also be erlang code but the scripts for the users can be any interpreter which can talk to the api server.the workspaces will have a paint workspace, an email and im workspace, editor, etc. all very full applications, and thus consistent. (simliar to an internet appliance or palm like as u said, with less diversity as in usual OSes)I thought of converting everything into one file format, that would be very rich something like flash is. perhaps using svg or something similar (this needs more thinking). that means that even as you browse the net, you get the converted version to look at. this can avoid malicious code, and probably smarter to build a "browser" that simply has one main format.this format is used by (everything in the system but also) the editor, so to write an email, you first write your document in editor, then open the email app, which simply adds the necessary headers and prepares the mail, you write the receipient and add an attachment if you want and send.the program doesn't send it -I like the concept that the system provides the means and not everything bundled together in the same app.the email program just saves it in the right folder (or alternatively sets some field in a database, if you want to go the database filesystem way)there is a mail server that sends everything out.it is also possible to make all communication between users of this os secured (crypto) and compressed (bzip2 would be sufficient)okay, perhaps i had more ideas that I wrote somewhere but this is plenty enough for now. would love to hear more if you ever start developing this seriously. Nice to see more people thinking in this direction :)Thanks, kobi [k_lurie@gbrener.org.il]

You don't need a "unicode string object" to work with unicode in Erlang. It's lists of integers. UCS-4 is lists of integers. You're done.

Presumably your data goes in and out as UTF-8, and there are functions for converting that to lists of integers. If you're just concatenating UTF-8 or leaving it as-is you don't really need to worry about transforming anything. Latin-1 is of course obvious since the byte values are the code points.

If you're doing things like locale sort or locale based upper/lower case or weird codecs then yeah, you need some deeper unicode functionality... but there is an iconv driver in jungerl that you could use for that.

At least lists of integers is better than having UTF-8 and not knowing where the character boundaries are. However, I was hoping for more intimate knowledge of Unicode, such as a Unicode-aware regex library.

Anyway, I'm sure it can be worked out. The security infrastructure is probably a harder problem.

I randomly came across your blog while trying to find the answer to a simple Erlang question.

I'm fairly new to Erlang itself, but I'm learning it because it's interesting to me for just the same reasons you mention. I firmly believe that as processors get more cores, the importance of parallelization is going to increase to the point where existing systems are not going scale to exploit the full potential of the hardware.

To answer questions about security in an Erlang system, it is possible (and very straightforward!) to implement security mechanisms beacuse processes can only communicate with processes they know the PID of, or can get the PID of through other means (by getting another process to send it to them, or by name if it is registered). There's no shared memory, no global variables, etc. to allow for other forms of communication. So, by spawning a process, you have complete control over how they can interact with the rest of the system.

I often hang out on #osdev on Freenode as a-priori. You should come there sometime and talk about your ideas.

> To answer questions about security in an Erlang system, it is possible (and very straightforward!) to implement security mechanisms beacuse processes can only communicate with processes they know the PID of

What about Bob's comment that, "Acquiring such references isn't exactly easy, but they're predictable in current versions of the VM"?

It seems to me that it's better to have the messaging mechanism step in and provide access controls on all messages. It reminds me of UNIX process groups.

What about Bob's comment that, "Acquiring such references isn't exactly easy, but they're predictable in current versions of the VM"?

What he means is that the means to acquire the PID of another process are limited. If you spawn a new process, you are given its PID. If the process is "registered", then it can be referred to by name. Otherwise, you must be sent the PID in a message. Typically, a supervisor process will send its children its PID so they can communicate with it. So, a supervisor can control all access those children have to the rest of the system.

It's the registered processes that are the sticking point, since those names are global.

> Instead of using MINIX, I would recommend using L4. It's advantages include:

Interesting. I was checking out L4 recently. Of course, when you say, "Freedom from POSIX semantics", I'm guessing that'll make it really hard to port Erlang OTP. Remember the cool thing about Minix + Erlang is that it's something simple I can do today. The trick is to start moving parts of Minix into Erlang. If I start with L4, it's a much harder battle to get going, right?

> It seems to me that it's better to have the messaging mechanism step in and provide access controls on all messages...

Requiring a microkernel to check the access controls of each and every message passed is exactly the sort of wrong thinking that relegated Mach to oblivion. Well-meaning in principle, a nightmare on performance in practice.

> Requiring a microkernel to check the access controls of each and every message passed is exactly the sort of wrong thinking that relegated Mach to oblivion. Well-meaning in principle, a nightmare on performance in practice.

>> Instead of using MINIX, I would recommend using L4. It's advantages include:

>Interesting. I was checking out L4 recently. Of course, when you say, "Freedom from POSIX semantics", I'm guessing that'll make it really hard to port Erlang OTP. Remember the cool thing about Minix + Erlang is that it's something simple I can do today.

I've been thinking of doing this as well, you might want to look at OKL4 at http://wiki.ok-labs.com/The good thing about this version of L4 is that they have a Linux environment called Whombat that let's you easily port porgrams that will compile under a 2.4 kernel. I've successfully ported scsh over, so I'm sure with a little work OTP shouldn't be a problem, (in fact if any one is interested in collaborating...) The good thing about L4 is that it has fast message passing, light weight and no so light weight threads, and super fast context switching all ready for you. These are things Joe Armstrong has said are essential to build a good concurrent system.( http://www.se-radio.net/podcast/2008-03/episode-89-joe-armstrong-erlang )

The hurdles are:1) OKL4 is not SMP aware :(2) No DMA network support so you have to do a lot of work to get, say an e1000 or tg3 driver to work.3) the current release runs under QEMU but doesn't work under VMware.

Once you get it ported into Wombat, then you can begin the work to get it running as a real Iguana service. But, as I said above the current version of OKL4 cannot take advantage of multicore CPUS (yet).Still I think L4 will eventually be the perfect platform for an "Erlang Machine"

(I guess I should come clean and admit that I don't know my @ss from a hot rock, and I have no free time to actually accomplish anything of such scale. *sigh* If only I were a wicked sharp systems programmer with a ton of free time!)

"Unlike object-oriented systems today that have many objects and few threads, unify the concepts of objects, threads, and modules into something I call a "cell"... That means every object has its own thread."

Nice idea 'cept Minix 3 doesn't have virtual memory support (chmem'ng everything can be a pain, requires root if i recall), the erlang tar file wont extract properly due to Minix tar long file name issues, even if that issue were to be resolved I'm not 100% sure Erlang would compile properly unless gcc where a dependency since Minix uses amsterdam compiler kit by default, even if that were the case and Erlang were supported, why base this new initiative off an os that has basically zero nic/audio/graphic support from drivers? Don't get me wrong I love a microkernel idea but Minix 3 is going to get you neglected from the start...

Bob Ippolito said correctly that "acquiring such references isn't exactly easy, but they're predictable in current versions of the VM"? and Michael Melanson claimed that "what he means is that the means to acquire the PID of another process are limited...."

No. That is the ideal case. It is also true that pids (in the current erlang runtime) are sometimes guessable with a bit of work.

This means that you cannot build real security within the current erlang runtime that relies on processes only communicating with other processes if they have been explicitly provided with the necessary pids. Erlang would have to be improved, making local pids unguessable, for this to be practical.

I encourage you to have a look at my blog (taooferlang.blogpost.com) as you ideas about userland OS is what I was striving to achive with X Erlang. Everything in Erlang except bare minimum. Even scheduler is in Erlang. C routines talk to underlying OS exclusively through APR library which makes the system portable to dozens of OSes. Exactly single executable file, nothing else is needed to bootstrap the system, etc.

My thoughts after reading through this, yes I know it's three years after.. why build a new brand os? Erlang is still distributed and VM so, just build an os straight off. Linux could be use as base as could Windows Server 2008 Core?It simplifies things, and it's possible to start of directly. And about all this troubles with security and so forth, if we ever would develop a thing like this, it would advance Erlang quite alot in security I think =)

Hey, Josef, it's simple--I think Erlang leads to safer, more robust code than C. Hence, it's an interesting idea to start with a microkernel and write as much of the OS as possible in Erlang. Starting with Minix 3 isn't all that important--it's just that Minix 3 has many of the same goals that I would have, and I've already read the source code for the original Minix. Of course, to be realistic, I think this is an interesting project, but I definitely don't have enough free time right now to attempt it.