Oracle Blog

2:5030/359 Roman Shaposhnik

Monday May 26, 2008

I still remember the coolness factor that the first fully functional and user friendly LiveCD with Linux had.
From a technical stand point it wasn't really that much of a novel concept (all major Linux distributors
had a capability of booting into a function Linux kernel for the installation and troubleshooting purposes)
but as everything with Linux -- it had some significant social implications. What
Klaus Knopper singlehandlely changed was
not what Linux was capable of, but rather how it presented itself for the first time. All of a sudden just about anybody
could take it for a spin. It got friendlier, it got less intrusive and even if you didn't like it -- hey, at least it
hadn't made you slave away for a day installing it (and trashing your Windows partition along the way). Fast forward
to 2008 and you can see that one of the most prevalent Linux distributions, Ubuntu, still uses ideas from Knoppix
to advance Linux adoption. Life is good, if only... boring. Yep, that's right -- there's nothing exciting about
a Linux LiveCD from major vendors anymore. It is polished, mature and leaves nothing to blog about. Not to worry
though, with the Project Indiana debuting its live CD
of OpenSolaris a couple of weeks ago we now have a brand new frog to dissect and marvel at.[Read More]

Friday Aug 24, 2007

Long time ago in the galaxy not so far away when you had to upload a
file to your friend you would call him up and have a ZMODEM session.
With this new fad called Internet it is all about SSH of course.
But is the default configuration secure enough? Is creating an account
for a friend who needs to upload a file to you all you need? How do
you communicate credentials?
[Read More]

Friday Jun 08, 2007

I'm a fundamental believer in scratching your own itch to be the
best kind of motivation for software development. And
Project D-Light
is not an exception here: if I want it to be
useful for others I have to make it useful to myself. Of course,
given that Project D-Light tries to utilize the most cutting
edge features of the DTrace technology it just makes sense to
run it on the most recent build of Solaris. And the fact that
we are building a tool first and foremost for the developers
pretty much narrows the choice to the Solaris Express Developer
Edition (SXDE 5/07).

Of course these days I do most of my development on my
laptop (good old ThinkPad T43) but the Solaris OS that I have
there is kind of clunky. It is a heavily tweaked Solaris 10,
which serves more as a proof that it is possible to run Solaris 10
on a ThinkPad after all, rather than a convenient development environment.
In short I had all the reasons to upgrade when I settled on a quest
for the ultimate development environment based on Solaris OS. The rest
is my account of this quest. You've been warned ;-)[Read More]

Friday Oct 20, 2006

Most of the time when you write your C or C++ code on a modern
UNIX platform you don't give much thought to the strict compliance
with various standards which used to rule the disjoint UNIX
landscape some odd number of years ago: XOPEN, POSIX, BSD, SVID, etc.
And if an application has only the very same platform you're developing
on as a deployment platform all is well. That doesn't happen all that
often though. And once you face the neccessity of making sure that
your application behavior is preserved across the variety
of UNIX-like OSs there the fun begins. The biggest problem is that
one of the cornerstone pieces of functionality every application
on UNIX depends upon has the most colorful history behind it. Of
course I'm talking about libc and the number of (sometimes conflicting!)
APIs it has to offer in order to comply with various standards.

Now, not a lot of folks know this, but libc is highly configurable
at an application compile time. If you have a reasonable compiler and a reasonable
libc implementation you can ask libc to behave like it used to during
the XOPEN times, BSD times or SVID times. The way you do that is
usually via specifying macros like: POSIX_SOURCE, etc. Here's a complete
list of macros you can tweak from the latest revision of Glibc:

And it should be quite straightforward to realize that once you add something
like this to your compilation line you're essentially changing behavior of
quite a few APIs (and YES! things can break or stop working at all):

$ cc -D_XOPEN_SOURCE=500 application.c

What is much less straightforward is the fact that vendors slice and dice what
they give you by default at their pleasure. For example GCC always defines
_GNU_SOURCE which means that by default you're getting much more non-standard functionality than would
otherwise be available (and yes! sometimes getting more than you've asked
for is a portability nightmare). And when you move your application to a place
where the defaults are different you have to fish out for the interfaces you
need and how to enable them. What's even more important is that you have to constantly
be on a lookout for different standards implementing some of the interfaces a
little bit differently. Which means that if vendor A decided to give you an API
which happens to be part of the larger standard by default, there's no guarantee
that when you go to a platform where that single API is not available unless you
explicitly ask for the standard it is part of other thing won't break.

Now, personally, I got bitten by it when I was porting an application from Solaris
to Linux and got bitten by the fact that a very simple and unassuming call to
sigset(2) was, in fact, part of the larger XOPEN standard and was not available
on Linux by default unless you ask for it. On Solaris, however it is. So the
dilemma I was faced with was either to augment my compilation line with
something like: -D_XOPEN_SOURCE=600 (which I knew would break a couple of
other things) or fish for something else that would only enable the sigset.

Wednesday Jul 26, 2006

Time and again I tell all my friends who happen to develop
for UNIX-like OSes to ditch the man pages and always read
POSIX standard
instead.

Case in point -- just a couple of days ago I had to make an internal
library behave on Linux. The reason it did misbehave was because
of the following code snippet:

pthread_create(&id, NULL, do_stuff, NULL);
workers[id].started = 1;

Of course, on sensible systems even the manpage for pthread_create(3)
tells you that you shouldn't be expecting anything from id.
It is, after all, an opaque datatype 'pthread_t'.

However, on Solaris it also has an additional property of being
akin to PID. IOW -- an ever increasing integer counting your threads.

Now, don't get me wrong -- as the title suggests if it ain't covered
by POSIX don't expect anything from it. However, the more I think about
it the more I like the way Solaris does it. And here's why:

I believe its to be a good programming practice to always make it
easier for consumer to enumerate objects coming out of producers
like pthred_create. It is easy enough for me to do on Solaris, but
on Linux I have to resort to keeping track of these objects
myself -- I need a global counter, locks around it and so forth.
On Solaris -- it's just an index.

Now, the reason they did it the other way on Linux (or better yet in glibc)
seems to be that they wanted to simply return the memory address of
the actual datastructure representing one particular thread. Nothing
wrong with that except that small ever-increasing integers are kind
of like addresses anyway, but they have an additional advantage
of being limited to an "address space" of an array which
lets them cross the border of different address spaces much easier.
Without increasing a complexity on the consumer side. And that's
a biggie in my opinion.

Saturday Jul 22, 2006

When I was a postgraduate student at St.Petersburg State University
I had come across the writeup from the Tom Duff (yes! of the
Duff's device fame)
where he stated that shared libraries are pure evil one true
sign that apocalypse is at hand. At the time I didn't gave it
much thought, but now that I've worked for Sun for some odd number
of years I think I tend to agree with him.

I believe that the main gripe I now have with shared libraries (AKA
Dynamic Shared Objects -- DSOs) is the fact that they truly aim
at solving two mutually exclusive problems: give vendors a flexibility
to patch systems "live" and also protect end-users from experiencing
failures of the unsuspecting applications which don't want to be
patched.

One of the tools for protecting the endusers is, of course, versioning
of the symbols in DSOs
introduced by Sun more than 10 years ago. And even what Sun did was somewhat of an overkill,
but the GNU crowd decided to go the whole nine yard as far as complexity
is concerned when they decided to "augment" Sun's versioning strategy
with a couple of
things of their own.

Of course the best of it is: "The second GNU extension is to allow multiple versions of the same function to appear in a given shared library."

Why do I care? Well, primarily because the following doesn't really work
as expected on Linux:

In fact it breaks. Horribly! Why ? Well, because pthread_cond_signal
happens to be a versioned symbol with the previous version still
available in glibc (and in libpthread.so, but that's a different story):

And regardless of the fact that the default one is supposed
to be the GLIBC_2.3.2 one when I call dlsym() I get the older
guy. Of course the older guys now has problems working with
a cond. variable initialized by the unitercepted (2.3.2)
pthread_cond_init and the whole thing goes kaboom.

Which means that in order for my code to work not only do I have
to now version my symbols in order to intercept only what's
needed but I also have to do a funny dance around dl[v]sym.

Versioning my own symbols was a bit of a challenge as well. Don't
get me wrong -- the Sun way of writing linker map files worked
quite nicely, but I really wanted to experience some of that
magical world of GNU asm:

and it took me a while to realize that the claim they make:
"This was done mainly to reduce the burden on the library maintainer."
is a bit further from realiaty than I expected -- you still need
the mapfile!

Tuesday Jul 11, 2006

Once upon a time there was a small project called GNU, and they wanted to build a system
which wouldn't be UNIX but rather a free ad lib giving every human being a power to tinker
with a capable computing environment. Their goal was a noble one and they started off
with building one cornerstone tool for building everything else – the C environment. And
thus Gcc and Glibc were created. Of course, because of the evil spell of a particular
committee Glibc had to suffer a multiple personality disorder from the day it was born. Because
you see, every libc has to serve two masters at the same time by being a pure language
library (just like, say, C++ standard library) but also by being an interface to the underlying
OS take care of the system calls. Of course, since the GNU project didn't really want to
have a UNIX kernel at their core GNU libc had to work with HURD. And so it did. Faithfully.
And it was a nice piece of software (that is to the extent anybody can call GNU coding
style nice).

Then 1991 came a long and Linux was in, that is – shopping for a suitable C library to suit
its kernel needs. GNU libc, of course, was an obvious choice not only because it was GNU
software, but because it was paired up with gcc – a compiler of choice for early Linux developers.
Of course, nobody wanted to deal with triple personality disorder (Glibc would have to
support two kernels at the same time) so they forked. And both pieces of software remained
to be pretty good. The only problem was – Linux kernel developers were not really interested
in developing the language personality of libc: after all they “don't do userspace”. So it wasn't
really a surprise when around 1997 they determined that it was easier to add support of Linux
kernel into the vanilla GNU libc (and chage the version from 5.X to 2) than to add all those
features of GNU libc back into their fork.

And thus a monster of complexity was born. Because by that time Glibc was really going out
of its way to support everything and the kitchen sink. All of the UNIX standards fashionable
at the time and on top of that a bunch of dubious extensions. The reasoning was simple: “because we can”.

At that time I still felt for GNU libc (even though personally I do believe in “small is beautiful”)
at least they had their excuse for being as bloated and complicated (does declaring a function
really need to require 32 lines of code and 4 different macros?) as they were. But it all came
crumbling down when I read this post from a raving\^H\^H\^H\^H\^Hoppinionated GNU libc
project lead Dictatorship of the Minorities.

So, I figured, if GNU libc is now officially a Linux only project may be its time to clean it
up or just throw it away and replace it with the proper C99 compliant libc which wouldn't
be rated “M” for the purposes of reading the code ? Like a
Solaris libc or a
Plan9 libc or an
Mplayer libc or a
BSD libc – anything but the stuff that has incorrect C99 code in its headers.

P.S. In the ideal world, of course, libc would be also purged from a split personality disorder
and made into two libraries – one for supporting system calls and system aspects of POSIX
and the other one for supporting C language. The later will have to be shipped with a compiler
and made as fast as possible using things like IR inlining. Oh well, a topic for a different
post I suppose.