Re: A comment about changing kernels

From:

Jonathan S. Shapiro

Subject:

Re: A comment about changing kernels

Date:

Thu, 27 Oct 2005 20:37:12 -0400

So we come finally to the question: *if* the choice is between L4.sec or
Coyotos, which one?
I have known this question would come, and I have very mixed feelings
about answering it. I think it is fair to say that I had a significant
role in the design of L4.sec (though not in its subsequent evolution),
and I want both L4.sec and Coyotos to succeed.
First, let me suggest that the question needs to be expanded. I think
there are three questions that need to be evaluated:
1. From a technical perspective, which kernel is better for the Hurd?
2. From a human perspective, which group will work more actively
with the Hurd team to support the use of their kernel?
Which development practices are more compatible with the Hurd and
GNU approaches?
3. Where does the design experience exist in this kind of system.
I. Selection of Kernel
Which kernel to use, in my opinion, is not really clear. Neither group
can say with confidence when their kernel and core system will be ready.
Here are the issues that I see for L4.sec:
1. Need for COPY. In order for this issue to be resolved, two things
are necessary:
1.A A COPY operation must become available in L4.sec
1.B. It must be detectable whether a capability is severable
(that is, whether any upstream REVOCABLE COPY has been
performed).
2. Need for a complete persistence design that does not violate
the requirements for "suspicious collaboration".
-- This is a design documentation task. It may or may not
expose any architectural issue. Whether it does is what
needs to be learned.
Both of these issues *can* be addressed in L4.sec. The first is purely
an implementation issue; there is no deep design problem here that I can
see. The second is the one that worries me, because it has implications
for resource DoS.
IF these issues are addressed, then the choice becomes a very close
thing, and the selection of kernel is probably not critical at that
point. However, this is a very significant "if".
Advantages of L4.sec: there are more people working on it.
Disadvantages of L4.sec: the conceptual jump from L4 to L4.sec is
surprisingly large, and it will be many years before they
have fully come to appreciate the implications of it.
Advantages of Coyotos: while the code change is large, the conceptual
jump is very small. We aren't making any basic paradigm change.
Disadvantages of Coyotos: small group.
Both groups now have commercial contracts as well; there is no
expectation that either one will go away.
II. Working Style
For many years I have attempted to get "inside" the L4 design process.
Hermann has tried to help. With the UNSW group this has somewhat
succeeded. With the Dresden and Karlsruhe teams it has consistently
failed. Even while I was the editor of the L4.sec specification, I could
not get them to engage in discussion. Certain individuals, most notably
Espen and Kevin Elphinstone, have worked very hard to overcome this, but
for the group as a whole this is a very old and comfortable behavior
pattern, and I don't think that it is likely to change. Neal may wish to
add his own thoughts on this, or he may not.
The problem goes two ways. While the EROS and Coyotos lists have always
been open, relatively few of the L4 group ever joined it. Jochen,
Hermann, Gernot, Espen, and Kevin *did* join at various points. Some
others have as well.
My group takes a different approach. We simply don't worry about people
stealing our work, because we are too far ahead in our area for that to
be practical. In some sense, because we are a research group, we
actually *want* our work to be stolen! Also, we are very willing to
share credit broadly. These values have allowed us to adopt an attitude
that I jokingly refer to as "developing naked in public". Aside from the
occasional discussion in a hallway, we do ALL of our design discussions
on the lists. We want to learn about our mistakes as early as possible,
because we believe that in the area of ideas, "infant mortality" is a
very good thing. We want bad ideas killed as fast as possible.
I would say that this approach has been an overwhelming success. Stop
and ask yourself just how L4 turned into a capability system...
We also want to benefit from the broader experience of our mentors. If
you look at our list archives, you will find comments there from just
about everybody still alive who has worked on capability systems.
So: the Coyotos group is more transparent, but whether this makes us any
easier to work with in practice is unclear. We *do* engage actively and
constantly in the kind of design debates that have happened here for the
last month (though usually not quite at such a high bandwidth), and we
have had many people come over the years and say to us things like "I
learned more from those archives than from my OS textbook."
One thing is very similar about the two groups. They are both led by
very experienced OS architects. Hermann's CV is very very impressive.
Mine, I think, is respectable. Both of us have delivered real OS's in
real applications in the real world **multiple times**. Each of us has a
view of how to proceed that is justified by experience, and both of us
emphasize design ahead of code. Code done wrong is just waste, and
operating system code is depressingly easy to do wrong. Kernel
development is the most challengingly overconstrained implementation
domain in all of computer science.
Somebody here started a thread entitled "Let's do some coding :-)". I
understand why that individual wanted to do that, and it probably made
sense for what they were trying to accomplish, and it seems to match the
FLOSS attitude in general. In the Coyotos effort the code generally
comes last and we rarely need to write it more than once. We prototype,
of course, but we do not really believe in iterative improvement in this
particular design space. When you screw up in a security kernel, it
isn't because you wrote bad code. It's because you missed a design
constraint. Changing a significant design constraint usually means
rewriting a very big chunk of stuff. Hermann, I suspect, feels much the
same way about L4.sec.
I mention this because it is very definitely NOT the usual FLOSS way of
doing things, and it is going to be a source of frustration to people
here in the early stages. I can also tell you that my group, at least,
will not change this practice. Our goal is a kernel for critical
systems. For general purpose software, I agree that the "code last"
approach simply doesn't work in practice. For small critical systems
like this one, it has been demonstrated in practice to work VERY well.
As to participation and engagement with the Hurd project, well, I don't
see anybody here complaining to Richard that Espen is asking too many
disconcerting questions. Depending on your point of view, this may be a
good thing. Espen, I must add, is a *terrific* guy! But where has the
*rest* of the L4.sec group been for the last month? I know that *all* of
my group is now on this list. They have been quietly helping me decode
some of the questions, and Eric Northup has surfaced here once or twice.
III. Design Experience
If the Hurd group chooses to try to incorporate capability ideas, you
are going to need an "old man" -- somebody who has a lot of experience
in this style of system building who can answer questions and help keep
the design issues untangled. As all of you can see by now, this design
space is very different from the one you all learned in your OS courses.
It is much more precise, often takes different approaches, and proceed
much more from principles and invariants than conventional OS design
does.
One reason that I have been able to sustain this discussion is that I
had a great mentor. Poor Norm Hardy spent *years* straightening me out
on this stuff, and he had help from Charlie Landau and Bill Frantz.
Whichever group you decide to work with, you will need somebody to play
that role.
And this is where I am going to get in trouble with the L4.sec group:
I'm not aware that they have anybody who can fill that role. Actually, I
had hoped to fill some of that role for *them*, but the communication
boundaries have been too high.
> However, what I would really like to hear from Shapiro is what's _wrong_ with
> Coyotos. If anyone knows, it would be him. :-)
I'll answer this shortly, but under a separate subject.
> Also, if we decide to use Coyotos, will he continue to make it his full-time
> job to hang around on this mailinglist? Or is that just a PR move to convince
> us? ;-)
> I would consider it a nice bonus to have a verified kernel. :-) But I don't
> think this is very important in this decision.
I agree, and I think that the Hurd group should completely ignore this
issue.
shap