So you have taken the test and you think you are ready to get started with OS development? At this point, many OS-deving hobbyists are tempted to go looking for a simple step-by-step tutorial which would guide them into making a binary boot, do some text I/O, and other "simple" stuff. The implicit plan is more or less as follow: any time they'll think about something which in their opinion would be cool to implement, they'll implement it. Gradually, feature after feature, their OS would supposedly build up, slowly getting superior to anything out there. This is, in my opinion, not the best way to get somewhere (if getting somewhere is your goal). In this article, I'll try to explain why, and what you should be doing at this stage instead in my opinion.

The importance of design

First, heavy reliance on tutorials is bad for creativity. More often than not we end up blindly following the thing without fully understanding the verbose paragraphs describing what we're doing, no matter which big red warning the author has put against this practice. What do we get this way? Many low-level tutorials on the web describe how to create a UNIX clone mainly written in C with a monolithic kernel and a simple console shell. Reminds you of something? That's normal, we already have tons of these around. It's quite sad to spend all this time in OS development only to get yet another UNIX-like kernel with more limited hardware support than the existing ones, in my opinion, though you might think otherwise.

Another reason for not following this approach is that having our head stuck in code makes us short-sighted. There are many pitfalls which can be avoided through some time spent designing our thing before coding. Simple examples : asynchronous I/O and multiprocessing. There's nothing intrinsically hard about supporting those, given sufficient doc on the subject, if you've made the decision to use them early enough and stick with it. It's just a different way of thinking. However, if you started without those and try to add them to your codebase later, the switching process will be quite painful.

Finally, the last reason why I think that design is important even for hobby projects is that their teams are small. Generally one or two developers. This is not necessarily a bad thing : in early stages of an OS project, you generally don't want endless arguments and heaps of code snippets written by different persons in different ways. The obvious drawback, however, is that unless you focus on a limited set of things and try to do these well, you're not going very far. This is why I think at this stage you should be not be writing code already but rather defining what the project's goals are, where you want to go.

Exploring motivations

To begin with, forget that you're targeting a specific computer, and even that you're writing code. You, as a team of 1-2, maybe more human beings, are designing something which will provide some kind of service to other human beings (which may simply be the members of the team). It all begins here. What is this service which you want to provide? When will you be able to say that your project is successful? When will you be able to say that you're not on the right track? To encounter success in a hobby OS project, like in many other kinds of personal projects, the first step is to define it clearly.

So go ahead, take a piece of paper or some other mean of text storage (in my case it was a blog), and explore your motivations. If you're writing an OS because you have some gripe with the existing ones you know of, spend some time finding out what you don't like in them. Try to understand why they are so, explore their evolution in time, read books on the subject if you can find them. Even if it turns out to be a random choice, you've found something : the importance of not doing this part randomly, and to pay particular attention to its design. If you're writing an OS in order to learn something, try to define what you want to learn in more details. Maybe you want to know where the process abstraction comes from? Through which process does one goes from a complex and dumb electrical appliance which does calculations to something which can actually be used by human beings without a high level of qualification?

In all the cases, take as much notes as possible without disturbing your thinking process. There are two reasons for this : first you can review said notes later, second translating thoughts in text forces you to be more rigorous than you would be if you could stick with ideas so vague that they can't even be directly expressed in English (or whatever language you may fluently speak). Try to be as precise as if these notes were to be published and read by other people who are not in your head.

Defining a target audience

After having defined why you're doing this, try to quickly define what your target audience is. An operating system is an interface between hardware, end users, and third-party software developers, so you have to define all three to some extent : what hardware is your operating system supposed to run on? Who is supposed to use it? Who is supposed to write software for it, and what kind of software? You don't have to be overly precise at this stage, but there are some things which you'd like to decide now.

On the hardware side: you should at least know how the users will interact with the hardware (Keyboard? Pointing devices? Which sort of each?), what range of screen areas you wish to cover, how powerful the oldest CPU which you want to target is and how much RAM is guaranteed to be there. You might also want to know right away if your OS requires an internet connection to operate properly, what sorts of mass storage devices you're going to use, and how third-party software (if there is some) will be installed.

Quickly check that the kind of hardware you want to code for is homebrew-friendly. Most desktops and laptops are (with maybe the upcoming ChromeOS notebooks as an exception), video games consoles are a mixed bag (Nintendo's ones often are, others generally aren't), products from Apple and Sony aren't until they are deprecated. A word of warning about cellphones: though they may sound like an attractive platform to code for, their hardware is generally-speaking extremely poorly documented, and you should keep in mind that even when you find one which does allow homebrew OSs to run and has sufficient documentation about it available on the web, the next generation is not guaranteed in any way to offer such comfort at all.

The end user is perhaps the one thing which is hardest to define precisely, except when the developers work only for themselves. You should probably keep around the description of a number of stereotypes representing the kind of people which you want to target. Some examples: "70-year old grandma who has been offered a computer by her grandchildren for Christmas. Never used one. Pay special attention to her sight problems.", "40-year old bearded sysadmin on a company's network. Won't use a computer if it can't be operated in CLI mode with a bash-like syntax. Wants to automate many tasks which are part of his job, think about powerful scripting engines. Needs multi-user support, and a special account to rule them all which only him may access.", "20-year old creative girl who draws things using a pen tablet and various free software. A computer is a tool like every single other for her, it shouldn't get in the way more than a block of paper."

Third-party developers; first thing to note is that if you want them to get interested in your project at some point, you'll first have to attract them as end users anyway. People don't write code for a platform which they wouldn't like to use, except when they are paid for it. On the other hand, devs have a special status in that, to state the obvious, they create software. One of your tasks will hence be to define which kind of software you let them create.

It can range from them writing vital parts of your operating system for you (frequently happens in the world of FOSS desktop operating systems) to every software being written in-house and third-party devs being non-existent (has grown out of fashion these days, except in some categories of embedded devices). Between these extreme situations, there's a wide range of possibilities. Here are a few...

The design/management of OS parts is proprietary, but the spec is disclosed, so that third-party developers are free to implement or re-implement parts of it the way they want

Most of the operating system itself is made in a proprietary and undisclosed fashion, but you're open to third-party drivers and low-level tools (Windows 9x, Mac OS)

Same as above, but you ask that applications needing access to some low-level or otherwise "dangerous" functionality go through some approval or signing process (Recent releases of Windows, Symbian)

You don't tolerate third-party low-level apps and will do everything you can to prevent their very existence, but you offer a native development kit for user applications with generous access to the system APIs

Same as above, but native third-party user apps are considered dangerous and must go through some signing/approval process (iOS)

No native third-party app, everything goes through managed apps which only have restricted access to system capabilities (Android, Windows Phone 7, most feature phones)

With all this information in mind, you should start to have a fairly good idea of where you're heading, and be ready for the next step.

Goals and frontiers

Now that the idea is there, try to put it on paper. Describe what your OS is supposed to achieve, for who, on which hardware, who will code what, and when you'll be able to tell that the project is successful according to its initial goals, that it has reached the 1.0 release, so to speak. By defining the project's goals you define criteria for objectively measuring its success and failure, which will prove to be a very valuable resource later, be it only as a mean to avoid feature bloat and other ways of wasting your very precious development resources.

In the end, look at this global view of the project you have now and ask yourself this simple question : "Do I think I can make it?". If you think you should maybe lower your expectations a little bit, now is the right time, because the later you do it the higher the damage done. Play with all this stuff you have gathered together, polish everything until you have a project you're satisfied with (or maybe drop this idea of writing an OS altogether if you don't feel like it any more), and then you'll be ready for the next step, namely designing your kernel.