It's rather obvious that an update here is a bit overdue,
so here's some of what's been going on with me.

I spent the end of January trying to ensure that all of
the
stuff I thought was important for the upcoming Mac OS
X release was taken care of. That included
un-frameworkifying OpenSSL so that it's a
library, which
I didn't get to in December before my vacation, and
making sure PHP builds on
Darwin properly. I had
still found problems with the Darwin support I has
added to GNU
libtool, but by the end of the month, I had
that well enough in hand that PHP is now in pretty good
shape on OS X.

I think I pretty much suceeded at getting my high
priority
items taken care of, so that put me pretty much done
with Mac OS X for this release. Which was good timing,
because Friday, Februrary 2nd would be my last day at
the
company. A lot of people slack off when they know they
are
quitting, but I was exhausted by Friday; I barely had time
to pack up my stuff. Call me nuts.

I'd met Rohit Khare and Adam Rifkin almost a year
ago,
I think. They were in Silicon Valley trying to get some
things done for a startup that they had going up in
Seattle, and they came by to offer me a job. Now I got
job offers pretty often for whatever reasons; recruiters
called by office every week or so and I *always* turned
them down. I'd listen sometimes, because it's prudent
to know what's out there, but I like Apple and I really
liked working at Apple, and I certainly liked working on
Mac OS X and so that was that.

But Rohit is a blustery sort of fellow, and he talks
quickly for long stretches and so I got inundated with
information within minutes of meeting him--he came
with a PowerPoint presentation and everything--and
amazingly enough, he was making a lot of sense, and I
though he really had something interesting there. Even
so, I had to
pass, because I wasn't finished with my work, and I
have this compulsion to finish what I start, especially
when I think it's helping make the world a better place,
as Apple folks like to do.

However, many blustery conversations later,
and having approached The Big Milestone of helping
release a system I will undoubtedly prefer over any
other for a long time to come, I was ready to take a leap
into something new. It was probably the harder "life"
decision I've ever made, and honestly I was quite
terrified after having made it. I was, after all, giving up a
job I really liked for one I hoped I'd like even more, and
that's not a trivial thing--for me, anyway.

So Monday, February 5th was my first day at
KnowNow.
It took them a while, but they sucked me in. When there
is something meaningful to say about what we do, I'm
sure we'll have plenty to say. At the moment, I'll just say
that my first two weeks have been pretty cool, and I like
the people. That is, the terror is gone. Though would
probably make our receptionist like me better if people
would stop calling and sending email; she's
had a busy week.

That said, I got a lot of encouraging email of late.
Some
from MIT buddies, ex-Disney Online folks, family, and
people I've met since working at Apple. Many from
people I haven't met. My apologies for not responding
to all of them, but I have a new job and I should
probably do some work. I do read and
appreciate it, though.

Last week which was rather quite still, I was fillding
with
mod_perl and found out that I
was stomping on CFLAGS
in the mod_perl build in a way that that normally causes
things to
break, but in mod_perl's case was building find but
causing it not to build with -DEAPI like httpd, which is
really very bad. So I fixed that.

Chuck Murko got a
port
of cscope to Darwin
started, and I did a little more tweaking and checked it
into the Darwin CVS repository, where Umesh tweaked
it some more, and I think we have a nice, clean port of it
now. Chuck is making sure the resulting patches get
sent upstream for integration. (Thanks, Chuck!)

This week, since I do web things now, I was in the
market for a good open source HTTP library. I need
one that lets me specify the headers to stick into the
request and get the result back as a stream as the
server outputs it. So I looked around and the only one I
found was W3C
Libwww.

Libwww has all the features I want and a bunch
more,
so I download it and I wrote a really simple client which
sends a GET request. In this case, I don't care about
the response, I just care that some CGI program gets
poked. Libwww is a horrible library. The docs are
obtuse, the API is inconsistent, and were it not for the
example, I'd have taken a long time to write this very
simple thing. It should have been easy, but it was a
yucky experience. Plus I had to link in more libraries
than the number of Libwww functions I used in my
program. I know poeple put some effort into the
functionality, but that's some crappy library design,
guys.

So I asked around some more and someone at
MIT
suggested libcurl. Now this is
a really nice little library. Simple and functional. It does
exactly what I want with not fuss, and now I have a
happy program, and I'm a happy programmer. Libcurl
rocks.

Well, I got back last night from a 3-week vacation at home (Puerto Rico). Given I was at the beach three days ago in the not-very-cold-at-all Carribean, San Jose seems pretty darned cold. Of course, my buddy Eric, in town from Chicago, begs to differ. Vacations are good. I did my Christmas shopping rather frantically on the 23rd; an eight-hour marathon at the Majagüez Mall with my cousins did the trick. Went to the movies a lot (it's cheaper there)... Played dominoes a lot... And I rented a Cessna 172 and took each of my two sisters up for a trip around the island. Anyone know where I can buy Puerto Rican food in the Bay Area? I'm sure it can be done, but I'm terrible at finding such things...

In December I bought a digital camera, the Canon PowerShot G1. Not to plug a product, but that was one of the best purchases I've made (eBay, $800) in a long while. I have this rather expensive and quite excellent camera that I bought a long time ago, a Mamiya 7. It's a wonderful camera, and takes beatiful photos on 6x7cm film, but it's the short of camera one doesn't carry around all the time, and I tend to take few pictures with it. Plus, since I left the proximity of the MIT campus, I no longer have a darkroom to play in. So this new Canon G1 is great because it fits in my jacket pocket, and it's easy to use, and it's trivial and $FREE to get the pictures onto my Mac, whereas on my Mamiya, I need to get the film processed somewhere (in my case in Palo Alto), then scan them in. Needsless to say, I took a bunch of pictures in Puerto Rico, and I'm very happy with my new toy.

MacWorld Expo 2001 started today in San Francisco, with an impressive suite of new product annoucements at Steve's keynote. Everyone's wondering where they are going to find the cash to buy a new PowerBook G4... I'll be at the show tomorrow, and probably again this week, depending on what's going on. I should be dropping by the Mac OS X booth every so often, this that tends to be a hub for people I know. The Stepwise BoF is tomorrow, I'll be there for that, as usual. This time I won't be sick and hoarse (probably).

James Duncan Davidson did come by Apple to repeat his talk from ApacheCon, and it was, as before, and excellent talk followed by some lively Q&A. It was great to have someone else with a similar perspective on open source come and share ideas. Them, of course, James had to tell us about what he lives about Mac OS X and what doesn't work for him, which was immediately followed by some of the authors of the offending software showing him that he can, indeed, do what we wants with our tools. So that went well, too. :)

I've spent a bunch of time recently trying to clean up some BSD compatibility issues in Darwin. Specifically, there are a few BSD-ish libraries which we install as frameworks.

For those of you that don't know a framework is a directory bundle which contains a library, it's headers, and any associated resources such as images and documentation. The cool thing about frameworks is that everything associated with it is in one place; it's all in this bundle which can easily be treaed as a single object. This has some nice advantages over installing in a typical BSD layout: you put the library in /usr/lib, the docs in /usr/share/man, the headers in /usr/include; if you have image or text resources, those go in /usr/share, and so on. The BSD system has advantgaes as well, but it's more geared at managing the operating system, rather than at managing components.

Anyway, the upshot is that we've been mangling libraries into frameworks because frameworks are what we prefer in Darwin, but of course then you end up needing to build clients in a slightly different way when building on Darwin, which creates an annoying portability problem. So the new plan for now is that new stuff should be frameworks, but let's not monkey around with BSD/Unix/POSIX style software so that porting code is easier. We can't rule the world all at once, after all.

So that works is all in CVS now. Affected projects include libSystem (!), zlib, tcl, and kerberos. I still need to do OpenSSL, when I return from vacation.

Speaking of which, I'll be taking off for vacation tonight until the start of MacWorld Expo SF. If you go to the expo, don't forget to swing by the Mac OS X booth and see what's up. In the meantime, happy holidays, all.

Today, I'm back in California, having arrived yesterday from my ApacheCon trip.

But first, two weeks ago was BSDCon. I met up with a bunch of people there and synched up some ideas we've been kicking around for a while.

I mentioned before (18 Jul 2000) that I had been talking to Jordan about sharing mode code in real time between Darwin and FreeBSD. At the O'Reily Conference, I sat down with Jordan and Charles Hannum (from NetBSD) to talk about actually sharing repositories for code like /bin/cp and /bin/ls, which we can probably agree on a common implementation. The general consensus was that this is a good thing, probably doable with some infrastructure work and a bit of political ironing, but none of us has had the time to really dig into it.

So then comes Chris Coleman and the Open Packages idea, which is a similar idea, except for the ports collections, rather than the base system software. This is a great thing, because there is already some good technology in place which can be tweaked to work accross BSDs (and other platforms as well), and the ports are already pretty neutral ground, so consensus is generally easier (though not trivial) to accomplish. So I'm very excited about this, because success here might be something we can extend further down into the system software, which for me is the jackpot. (From the Darwin point of view, most of the BSD subsystem is already a "port," since we're already drawing the code from NetBSD and FreeBSD; if we can find a way to share code uniformly accross our systems, that's just too cool.)

To make things even better, Keith Bostic gave us a great keynote at the conference, and one of his questions to the community was "Why are we sharing the code to cp?", which got more people thinking about it as well.

Last week, I was in London for ApacheCon. We started with a Hackathon, where the ASF members and some guests got together to work our some issues while we have everyone in one place. We had about 30 people sharing a 56K net connection, which was rather interesting. Mostly it was a good chance to meet everyone before the conference and hack out some code with all of the relevant experts nearby. I spent a lot of time trying to wrap my brain around libtool, and thanks to some help from Sander (from Covalent) and a kick start from Sascha, I got libtool working well enough that PHP will build on Mac OS X correctly. And, as a bonus, it even works. Rasmus hooked me up with a PHP CVS account, so I'll probably be committing the fixes this week, and I'll need to send a new patch to the libtool folks.

ApacheCon itself went very well. Duncan gave a fabulous talk about how Sun came to open source Tomcat via Apache, and the hurdles they had to get past to make it work, which I'm going to try to get him to repeat at Apple some time. It's a great success story for a big company starting an open source project well. Douglas Adams wrapped up the conference with a fun talk about the unpredicatable evolution of computing. For such a young conference (this was #3), it's going very quickly, and it's packed with good talks and all the right people there.

While we were there, I got to check out London with some of the ASF guys. Roy and Greg and I were staying in this small Bed-and-Breakfast type hotel, and we played tourist all over the city. I have to say I really liked it. The weather was pretty reasonable for walking about. The only thing that I didn't like was the itty bitty portions you get at restaurants (it's like I was on a diet for a week).

I've been getting poked at by a few people because I haven't updated my diary in a while.

One of the big things that has happened in the past month is that we shipped Mac OS X Public Beta, which is the first public release of the software I've been working on for the past three years. The reception has so far been very positive, and I'm pretty happy about the results. We still have a long way to go, but looking back, we've actually gotten a lot done.

One reason I've not posted here in a while is that I've been working on an internal project lately, to help with deploying Mac OS X at Apple so that more of the company is using it now that it's fairly stable. Unfortunately, it's not all that interesting to talk about.

One great thing is that the Darwin community is really picking up some steam. There has been a fair bit of code pouring in this month. Luke Howard is cranking on adding PAM support, which he started working on by back-porting the current linker to Rhapsody DR2, as all he had was a PC at the time. Fortunately, he now has a shiny new G4 do his work on. Check out his diary for some status. Rob Braun has been doing a bit of a code audit in parts of the system and has committed a large pile of fixes to clean up things from possible buffer overruns to nicer syslog output. Dave Z. is working on getting a Darwin 1.2 release out the door to coorespond with Public Beta. Rob's also started working on getting the FreeBSD libc I imported, which is the critical and most difficult software sync we have to do in order to get the rest of the system software updated. There has also been some work on creating a Darwin logo; I'm a big fan of Hexley.

Tonight I'll be driving to Monterey for BSDCon 2000, and Friday I'll be flying to London for ApacheCon 2000 Europe, so it'll be a good couple of weeks for conferences. I'll be repeating my USENIX talk at BSDCon; and I figure I'll get the PHP4 port finished up, which I've been putting off, as well as Apache 2.0 while I'm at ApacheCon.

Chris Coleman from Daemon News has been working for some time
on getting the various BSD distributions to collaborate on a
shared ports
collection, so that we can all benefit from each others'
work. This is a wonderful goal to strive for, so when he
called asking for some support from Apple, we agreed.

Note that Darwin is still at a stage where we are trying to
get the system itself built and released in a timely fashion,
which puts concerns about a ports system somewhat out in the
future, but certainly it's going to be important once we're
rolling along, and we would love to be able to take advantage
of the already existing infrastructure pioneered by FreeBSD
and expanded on by NetBSD. So while it might take some time
before we're actively using and contributing to the new
unified ports collection, David and I are both on the mailing
list and we'll try to make sure that we'll be ready to jump
in as soon as we're ready, so that Darwin and Mac OS X users
can all participate.

OpenSSH
somehow managed
to sneak itself into the Mac OS X build train (wonder how
that happened...); expect to see ssh support built into Mac
OS X Public Beta. A tip of the hat to the OpenBSD folks for getting us an
excellent ssh implementation under a free license, and to the
Unix
porting team that made the port to Darwin rather
straightforward.
Going forward, I want to start deprecating the use of telnet,
ftp, (especially) rsh and other clear protocols in Darwin,
replacing them with ssh, rsync-over-ssh, and so on.

Last week, I went to my first Burning Man festival. Now that's an
experience. The desert environment and weather was at times
quite brutal, but fortunately the attending citizens are
overwhelmingly supporting of each other, and there was much
borrowing and loaning of gear and supplies with our
neighbors, then off to explore the city and much fun. I went
for pretty much the whole week (took off Thursday and Friday
to pick up some more people), which was a good thing. Early
in the week, when the population is smaller and everything is
getting built up, it was much mellower. Apparently while I
was back in the Bay Area, the winds torn down some of the
cooler camps, so had I only come at the end, I'd not have
gotten to see them at all. Plus the weather in the latter
half of the week was pretty cold, even during the day.

I had hoped to fly in, but my flight school didn't like the
idea of dust all over the inside of their plane so much. I
saw about 20 planes there by the end of the week, though, so
it seems reasonable... I'll try harder next year. We camped
next to the Black Light Light Brite camp, which had
a giant Light Brite made of wood and lit up with UV lamps.
It was excellent, and the guys camped there the most helpful
bunch we could have camped next to. I had a blast, and next
year I'll be more prepared.

Earlier this week, I was at LinuxWorld, on the BSD BoF panel. There were plenty of questions, but it was rather uneventful overall.

The BSD code merge has been stagnating while I've been distracted by some other things. I need to get back to working on the libraries.

Some good news: OpenSSH finally works, thanks to a good catch from Melissa E. O'Neil.

I've been putting some work into GNU libtool in order to get PHP4 building properly on OS X. I've got a bunch of good changes ready in libtool, and a few in PHP, which got it to the point where it is compiling and linking without errors, but we're not linking in all of the files that get built, so you get a runtime linker error when you try to use it in Apache. Not sure whether this is a PHP makefiles thing or a libtool thing. Libtool's also not passing down linker flags like I expect, so that'll need some cleanup as well. I'm new to libtool, so I need to learn a few more things before I can move further. After php4 will come mod_perl, which I'm sure will have its own fun quirks.

There has been some very lively discussion on the Darwin development list about SystemStarter, which is a program I wrote to bootrap Mac OS X (it will be a replacement for /etc/rc). We're working out some issues about adding shutdown sequence support, some ideas about incremental startup, and parallelizing the startup sequence.

We finally finished renaming the Darwin servers so, for example instead of www.publicsource.apple.com, we're using www.opensource.apple.com, which is much easier for people to deal with (the old names still work, naturally).

I've been working on merging some FreeBSD code in the past couple of weeks. First, I pulled down FreeBSD's make code and /usr/share/mk suite, merged that into bsdmake in Darwin, and then tossed PB makefiles and got it building with its own makefiles. So now bsdmake builds with itself, so we know that it works for building BSD commands in Darwin. The goal here is to facilitate building BSD libraries without having to rewrite the makefiles, as I have been doing with the BSD commands. Munging the makefiles makes updating the code more troublesome, so I'd like to avoid that going forward.

I got libtelnet ported from FreeBSD as well, which was a good starting point, since it only has two clients in Darwin and it's static. Libc is probaby going to have to be next, which won't be as easy. Since it's going into the System framework, it can also build static, but making sure that existing programs don't all fall over will be difficult. Also, the fact that BSD keeps library headers separate from code (it's in include/, instead of lib/) makes it harder to keep things organized. We'll need some additional bsdmake/mk tweaks to get things going, and Chuck Swiger has been working on that some, which I'll take a look at next week.

I also talked to Jordan recently about giving people better access to the Darwin code, since right now you need to register to use CVS, even for the non-APSL parts of the repository. So I've been experimenting with a way to make the non-APSL parts visible anonymously via a side repository. We're also kicking around ideas for how to share code in "real time" rather than merging back and forth.

This week I'm at the O'Reilly Open Source Conference in Monterey. Yesterday I went to Prof. McKusick's tutorial on the BSD kernel, which was great, since I work almost exclusively outside of the kernel, and getting a better picture of what's inside is useful stuff.

Tonight, I'll be at the MacPerl BoF, so I can get a feel for what the MacPerl guys would like to bring forward to Mac OS X. The Apache beerfest is also tonight, but I'll probably miss it due to the BoF. Thursday I'm be on a panel on the Future of BSD, representing Darwin/Mac OS X.

I've gotten a lot of mail and feedback about my USENIX paper, mostly positive, some occasionally critical. Several people were hoping to find out different things, such as how administration of the BSD subsystem will be presented to the user. There are certainly many other interesting topics in the Mac OS X space and the relationship of the Mac OS user experience and the BSD underpinnings. Hopefully other people will find time to write papers on these various topics. I'd love to expand on the material in the paper I did write, and write up some new stuff altogether, but that takes a fair bit of time and effort. For the time being, I need to get back to other things.

Some in the Slashdot crowd apparently think I'm an idiot because I didn't spellcheck the paper. Actually, I did, but making lots of last-minute changes didn't help, I guess. Certainly I'm happy to correct such things, if for no other reason than my professors some college might read some of this stuff, but the goal was to communicate some issues we ran into, not to pass a writing test. My target audience seems to understand this, so I'll not lose sleep over it. My apologies for the typos and bad grammar in my diary; I know there are plenty of examples here as well.

Also some people think I'm being too generous to the original Mac development team. Now I don't program for Classic Mac OS--I never have--and that's largely because I find the APIs too bulky, I "grew up" in a POSIX world, and I don't like some of the fundamentals of the system. Those things are the reason we have Mac OS X in development, and why I have a job at Apple. That said, in talking to some of the people that really understand why things are the way they are, I did develop a lot more respect for the decisions that were made, and I do think that they were quite reasonable given the state of computing at the time. My friend Dan B. (as opposed to Dan R. who needs to turn his modem off of sometime so I can call him) pointed out that Amiga did manage to made preemption in the same timeframe as the Mac launch. That's certainly true, though I don't know how long each product was in development. But cooperative tasks could in theory be more efficient than preemptive tasks, and while the practical value of preemption seems obvious now, it wasn't necessarily a proven thing then. Memory was really the big issue, and the even Amiga übercomputer didn't protect memory.

One of the key points I was trying to make is that the priority set between the Unix and Mac worlds were very different. One a system where a one-to-one mapping between computers and users exists simply has a different notion of what "security" is than a system where many users share a computer, particularly if they do so simultaneously. it's not the case that security isn't needed in both systems, but the requirements are quite different.

Anyway, I firmly believe that the Mac team has nothing at all to apologize for, given their accomplishments.

I presented my paper on "The Challenges of Integrating the Unix and Mac OS Environments" at USENIX on Thursday morning. The session went really well. It was well-attended, and I fielded about half an hour of questions afterward. The general feeling I got at the conference was that many people in the Unix community are excited about the potential of Mac OS X.

Wednesday night was the "Night of the Living BSDs"; there were back-to-back Birds-of-a-Feather sessions for NetBSD, OpenBSD, FreeBSD, and BSDi, followed by a BSD coctail party. I met up with a bunch up people there, and it was a fun night. Jordan gave Mac OS X a nice plug at the beginning of the FreeBSD BoF.

Right after my talk, I flew to Detroit for Mac Hack, which I'll at now. This year's Mac Hack is packed with lots of sessions, and there is a lot of Mac OS X hacking this year. I'm taking the DropScript demo I whipped up for WWDC and wiring up a pack/unpack tool pair called "ShoveIt Deluxe". Pretty wimpy, really, but I only had one day instead of two, and I wanted to go to some sessions and see what everyone else is doing this year. I also ran into someone who was looking to hack up SystemStarter to display marching icons at the bottom of the screen as done in Classic Mac OS. It turns out he was asking about it nearby, and I overheard him mention some problems he was having (we don't public all of the API he needs yet), so I let him know I wrote SystemStarter and helped him out with it. I love Mac Hack; this stuff is just fun.

Josh and David and I spent some time in the past couple of weeks rewriting the hack we did last year, "Shagadellic Telly". Josh and Dave bough a "WinTV" TV tuner card and figure we could fix the "Win" bug in the name by writing a Mac OS X driver. We have a much cleaner driver this time. I wrote the Cocoa application (the Telly) which controls the driver and tells it where to blast onto the screen. This version, though still incomplete, works rather well, and has a bunch of neat features. The driver and app are in the "AppleBt8xx" driver project in Darwin, if you're ihterested in playing with it.