Let’s say you want to run a PHP application like WordPress under Apache. Here are the things you need to keep secure:

WordPress itself

All plugins, themes, customizations

All PHP libraries it uses (MySQL, image-processing, etc.)

MySQL

Apache

All libraries MySQL or Apache use: OpenSSL, libc, PHP itself, etc.

The kernel

All containerization tools

On Debian (and most of its best-known derivatives), we are extremely lucky to have a wonderful security support system. If you run a Debian system, the combination of unattended-updates, needrestart, debsecan, and debian-security-support will help one keep a Debian system secure and verify it is. When the latest OpenSSL bug comes out, generally speaking by the time I wake up, unattended-updates has already patched it, needrestart has already restarted any server that uses it, and I’m protected. Debian’s security team generally backports fixes rather than just say “here’s the new version”, making it very safe to automatically apply patches. As long as I use what’s in Debian stable, all layers mentioned above will be protected using this scheme.

This picture is much nicer than what we see in Docker.

Problems

We have a lot of problems in the Docker ecosystem:

No built-in way to know when a base needs to be updated, or to automatically update it

Diverse and complicated vendor security picture

No way to detect when intermediate libraries need to be updated

Complicated final application security picture

Let’s look at them individually.

Problem #1: No built-in way to know when a base needs to be updated, or to automatically update it

First of all, there is nothing in Docker like unattended-updates. Although a few people have suggested ways to run unattended-updates inside containers, there are many reasons that approach doesn’t work well. The standard advice is to update/rebuild containers.

So how do you know when to do that? It is not all that obvious. Theoretically, official OS base images will be updated when needed, and then other Docker hub images will detect the base update and be rebuilt. So, if a bug in a base image is found, and if the vendors work properly, and if you are somehow watching, then you could be protected. There is work in this area; tools such as watchtower help here.

But this can lead to a false sense of security, because:

Problem #2: Diverse and complicated vendor security picture

Different images can use different operating system bases. Consider just these official images, and the bases they use: (tracking latest tag on each)

nginx: debian:stretch-slim (stretch is pre-release at this date!)

mysql: debian:jessie

mongo: debian:wheezy-slim (previous release)

apache httpd: debian:jessie-backports

postgres: debian:jessie

node: buildpack-deps:jessie, eventually depends on debian:jessie

wordpress: php:5.6-apache, eventually depends on debian:jessie

And how about a few unofficial images?

oracle/openjdk: oraclelinux:latest

robotamer/citadel: debian:testing (dangerous, because testing is an alias for different distros at different times)

docker.elastic.co/kibana: ubuntu of some sort

The good news is that Debian jessie seems to be pretty popular here. The bad news is that you see everything from Oracle Linux, to Ubuntu, to Debian testing, to Debian oldstable in just this list. Go a little further, and you’ll see Alpine Linux, CentOS, and many more represented.

Here’s the question: what do you know about the security practices of each of these organizations? How well updated are their base images? Even if it’s Debian, how well updated is, for instance, the oldstable or the testing image?

The attack surface here is a lot larger than if you were just using a single OS. But wait, it gets worse:

Problem #3: No way to detect when intermediate libraries need to be updated

Let’s say your Docker image is using a base that is updated immediately when a security problem is found. Let’s further assume that your software package (WordPress, MySQL, whatever) is also being updated.

What about the intermediate dependencies? Let’s look at the build process for nginx. The Dockerfile for it begins with Debian:stretch-slim. But then it does a natural thing: it runs an apt-get install, pulling in packages from both Debian and an nginx repo.

I ran the docker build across this. Of course, the apt-get command brings in not just the specified packages, but also their dependencies. Here are the ones nginx brought in:

Now, what is going to trigger a rebuild if there’s a security fix to libssl1.1 or libicu57? (Both of these have a history of security holes.) The answer, for the vast majority of Docker images, seems to be: nothing automatic.

Problem #4: Complicated final application security picture

And that brings us to the last problem: Let’s say you want to run an application in Docker. exim, PostgreSQL, Drupal, or maybe something more obscure. Who is watching for security holes in it? If you’re using Debian packages, the Debian security team is. If you’re using a Docker image, well, maybe it’s the random person that contributed it, maybe it’s the vendor, maybe it’s Docker, maybe it’s nobody. You have to take this burden on yourself, to validate the security support picture for each image you use.

Conclusion

All this adds up to a lot of work, which is not taken care of for you by default in Docker. It is no surprise that many Docker images are insecure, given this picture. The unfortunate reality is that many Docker containers are running with known vulnerabilities that have known fixes, but just aren’t, and that’s sad.

I wonder if there are any practices people are using that can mitigate this better than what the current best-practice recommendations seem to be?

Share this:

There has been a lot of noise lately about the GOP health care plan (AHCA) and the differences to the current plan (ACA or Obamacare). A lot of statistics are being misinterpreted.

The New York Times has an excellent analysis of some of this. But to pick it apart, I want to highlight a few things:

Many Republicans are touting the CBO’s estimate that, some years out, premiums will be 10% lower under their plan than under the ACA. However, this carries with it a lot of misleading information.

First of all, many are spinning this as if costs would go down. That’s not the case. The premiums would still rise — they would just have risen less by the end of the period than under ACA. That also ignores the immediate spike and throwing millions out of the insurance marketplace altogether.

Now then, where does this 10% number come from? First of all, you have to understand the older people are substantially more expensive to the health system, and therefore more expensive to insure. ACA limited the price differential from the youngest to the oldest people, which meant that in effect some young people were subsidizing older ones on the individual market. The GOP plan removes that limit. Combined with other changes in subsidies and tax credits, this dramatically increases the cost to older people. For instance, the New York Times article cites a CBO estimate that “the price an average 64-year-old earning $26,500 would need to pay after using a subsidy would increase from $1,700 under Obamacare to $14,600 under the Republican plan.”

They further conclude that these exceptionally high rates would be so unaffordable to older people that the older people will simply stop buying insurance on the individual market. This means that the overall risk pool of people in that market is healthier, and therefore the average price is lower.

So, to sum up: the reason that insurance premiums under the GOP plan will rise at a slightly slower rate long-term is that the higher-risk people will be unable to afford insurance in the first place, leaving only the cheaper people to buy in.

This means there was a data error on the drive. But it’s worse than a typical data error — this is an error that was not detected by the hardware. Unlike most filesystems, ZFS and btrfs write a checksum with every block of data (both data and metadata) written to the drive, and the checksum is verified at read time. Most filesystems don’t do this, because theoretically the hardware should detect all errors. But in practice, it doesn’t always, which can lead to silent data corruption. That’s why I use ZFS wherever I possibly can.

As I looked into this issue, I saw that ZFS repaired about 400KB of data. I thought, “well, that was unlucky” and just ignored it.

Then a week later, it happened again. Pretty soon, I noticed it happened every Sunday, and always to the same drive in my pool. It so happens that the highest I/O load on the machine happens on Sundays, because I have a cron job that runs zpool scrub on Sundays. This operation forces ZFS to read and verify the checksums on every block of data on the drive, and is a nice way to guard against unreadable sectors in rarely-used data.

I finally swapped out the drive, but to my frustration, the new drive now exhibited the same issue. The SATA protocol does include a CRC32 checksum, so it seemed (to me, at least) that the problem was unlikely to be a cable or chassis issue. I suspected motherboard.

It so happened I had a 9211-8i SAS card. I had purchased it off eBay awhile back when I built the server, but could never get it to see the drives. I wound up not filling it up with as many drives as planned, so the on-board SATA did the trick. Until now.

As I poked at the 9211-8i, noticing that even its configuration utility didn’t see any devices, I finally started wondering if the SAS/SATA breakout cables were a problem. And sure enough – I realized I had a “reverse” cable and needed a “forward” one. $14 later, I had the correct cable and things are working properly now.

One other note: RAM errors can sometimes cause issues like this, but this system uses ECC DRAM and the errors would be unlikely to always manifest themselves on a particular drive.

So over the course of this, had I not been using ZFS, I would have had several megabytes of reads with undetected errors. Thanks to using ZFS, I know my data integrity is still good.

Share this:

I still remember vividly my first visit to Europe, back in 2010. I had just barely gotten off a plane in Hamburg and on to a bus to Lubeck, and struck up a conversation with a friendly, well-educated German classical musician next to me. We soon started to discuss politics and religion. Over the course of the conversation, in response to his questions, I explained I had twice voted against George W. Bush, that I opposed the war in Iraq for many reasons, that I did thought there was an ethical imperative to work to defeat climate change, that I viewed health care as an important ethical and religious issue, that I thought evolution was well-established, and that I am a Christian.

Finally, without any hint of insult intended, and rather a lot of surprise written all over his face, he said:

“Wow. You’re an American, and a Christian, and you’re so…. normal!”

This, it seems to me, has a lot to do with Trump.

Ouch

It felt like a punch to the gut. The day after the election, having known that a man that appeared to stand for everything that honorable people are against won the election, like people all around the world, I was trying to make sense of “how could this happen?” As I’ve watched since, as he stacks government with wealthy cronies with records nearly as colorful as his own, it is easy to feel even more depressed.

Based on how Trump spoke and acted, it would be easy to conclude that the “deplorables” won the day – that he was elected by a contingent of sexists or racists ascendent in power.

But that would be too simple an explanation. This is, after all, the same country that elected Barack Obama twice. There are a many people that voted twice for a black man, and then for Trump. Why? Racism, while doubtless a factor, can’t explain it all.

[Many Americans are] hurt, and they’re scared, and they feel like a lot of the United States just slammed the door in their faces.”

The status quo is not working for people.

Technocratic government by political elites is not working for people. Business as usual is not working for people. Minor tweaks to increasingly arcane systems is not working for people. People are feeling lost in bureaucracy, disaffected by elections that do not present a clear alternate vision, and depressed by a slow slide into increasingly dismal circumstances.

Government is not doing what we want it to do for us. And people are getting left behind. The left in the United States (of which I’m part) has for many years been very concerned about the way blacks and other racial minorities are systematically pushed to the margins of our economy, and how women are pushed out of leadership roles. Those problems are real. But the loss of jobs in the industrial heartland, the inability of a white, rural, working-class man to support his family the way his father supported him, the collapse of once-vibrant communities into poverty and despair: those problems are real too.

The status quo is not working for anyone except for a few lucky, highly-educated people on the coasts. People, honestly, like me, and like many of the other (primarily white and male) people who work in tech. We are one of the few beneficiaries of a system that is failing the vast majority of people in this country.

Russ is, of course, right. The Democrats have been either complicit in policies damaging to many, or ineffective in preventing them. They have often appeared unconcerned with the plight of people outside cities (even if that wasn’t really the case). And it goes deeper.

When’s the last time you visited Kansas?

I live in Kansas. The nearest paved road is about a 3-mile drive from my home. The nearest town, population 600, is a 6-mile drive. My governor — whom I did not vote for — cut taxes on the wealthy so much that our excellent local schools have been struggling for years. But my community is amazing, full of loving and caring people, the sort of people who you know you’ll be living with for 40 years, and so you make sure you get along well with.

I have visited tourist sites in Berlin, enjoyed an opera and a Broadway show in New York, taken a train across the country to Portland, explored San Francisco. I’ve enjoyed all of them. Many rural people do get out and experience the world.

I have been in so many conversations where I try to explain where I live to people that simply cannot fathom it. I have explained how the 18 acres I own is a very small amount where I am. How, yes, I do actually have electricity and Internet. How a bad traffic day is one where I have to wait for three cars to go past before turning onto the paved road. How I occasionally find a bull in my front yard, how I can walk a quarter mile and be at the creek on the edge of my property, how I can get to an airport faster than most New Yorkers and my kids can walk out the front door and play in a spot more peaceful than Central Park, and how all this is way cheaper than a studio apartment in a bad part of San Francisco.

It is rare indeed to see visitors actually traveling to Kansas as a destination. People have no concept of the fact that my mechanic would drop everything and help me get my broken-down car to the shop for no charge, that any number of neighbors or uncles would bring a tractor and come plow the snow off my 1/4-mile driveway out of sheer kindness, that people around here really care for each other in a way you don’t see in a city.

There are people that I know see politics way differently than me, but I know them to be good people. They would also do anything for a person in need, no matter who they are. I may find the people that they vote for to be repugnant, but I cannot say “I’ve looked this person in the eyes and they are nothing but deplorable.”

And so, people in rural areas feel misunderstood. And they are right.

Some perspectives on Trump

As I’ve said, I do find Trump to be deplorable, but not everyone that voted for him is. How, then, do people wind up voting for him?

The New Yorker had an excellent story about a man named Mark Frisbie, owner of a welding and fab shop. The recession had been hard on his business. His wife’s day-care center also closed. Health care was hard to find, and the long, slow decline had spanned politicians of every stripe. Mark and his wife supposedly did everything they were supposed to: they worked hard, were honest, were entrepreneurial, and yet — he had lost his business, his family house, his health coverage, everything. He doesn’t want a handout. He wants to be able to earn a living. Asked who he’d vote for, he said, “Is ‘none of the above’ an option?”

The Washington Post had another insightful article, about a professor from Madison, WI interviewing people in rural areas. She said people would often say: “All the decisions are made in Madison and Milwaukee and nobody’s listening to us. Nobody’s paying attention, nobody’s coming out here and asking us what we think. Decisions are made in the cities, and we have to abide by them.” She pushed back, hard, on the idea that Trump supporters are ignorant, and added that liberals that push that line of thinking are only making the problem worse.

I would agree; seeing all the talk about universities dis-inviting speakers that don’t hew to certain political views doesn’t help either.

And then we have a more recent CNN article: Where Tump support and Obamacare use soar together, explaining in great detail how it can be logical for someone to be on Obamacare but not like it. We can all argue that the Republicans may have as much to do with that as anything, but the problem exists.

“His supporters realize he’s a joke. They do not care. They know he’s authoritarian, nationalist, almost un-American, and they love him anyway, because he disrupts a broken political process and beats establishment candidates who’ve long ignored their interests.

When you’re earning $32,000 a year and haven’t had a decent vacation in over a decade, it doesn’t matter who Trump appoints to the U.N., or if he poisons America’s standing in the world, you just want to win again, whoever the victim, whatever the price.

…

According to the Republican Party, the biggest threat to rural America was Islamic terrorism. According to the Democratic Party it was gun violence. In reality it was prescription drug abuse and neither party noticed until it was too late.”

Are we leaving people out?

All this reminded me of reading about Donald Knuth, the famous computer scientist and something of the father of modern computing, writing about his feelings of trepidation about sharing with his university colleagues that he was working on a project related to the Bible. I am concerned about the complaints about “the PC culture”, because I think it is good that people aren’t making racist or anti-semitic jokes in public anymore. But, as some of these articles point out, in many circles, making fun of Christians and conservatives is still one of the accepted targets. Does that really help anything? (And as a Christian that is liberal, have all of you that aren’t Christians so quickly forgotten how churches like the Episcopals blazed the way for marriage equality many years ago already?)

But they don’t get a free pass

I have found a few things, however, absolutely scary. One was an article from December showing that Trump voters actually changed their views on Russia after Trump became the nominee. Another one from just today was a study on how people reacted when showed inauguration crowd photos.

So where does this leave us? Heading into uncertain times, for sure, but perhaps — just maybe — with a greater understanding of our neighbors.

Perhaps we will all be able to see past the rhetoric and polarization, and understand that there is something, well, normal about each other. Doing that is going to be the only way we can really take our country back.

The other day, this wound up with me singing a duet with my 7-year-old Oliver, and seeing a hugely delighted 10-year-old Jacob.

All last week, the boys had been lobbying for me to “be Santa”. They aren’t going to be able to be here on Christmas day this year, so I thought – why not let them have some fun. I chose one present to give them early too.

So, Saturday night, I said they could get ready for Santa. They found some cookies somewhere, got out some milk. And Oliver wrote this wonderful note to “Santa”:

That is a note I’m going to keep for a long time. He helpfully drew arrows pointing to the milk, cookies, and even the pen. He even started Santa’s reply at the bottom!

So, Saturday night, I snuck up to the attic, pretended to be Santa, and ate some cookies, drank some milk, and wrote Oliver a note. And I left a present.

Jacob has been really getting into music lately, and Laura suggested I find something for the boys. I went looking for something that could record also, and came up with what has got to be a kid’s dream: a karaoke machine.

The particular one I found came with two microphones, a CD player, audio recording onto SD card (though it’s a little dodgy), and a screen for showing words on any music that’s karaoke-enhanced.

Cue gasps of awe and excitement from the boys when we came down in our PJs and sweats at 6:45 Sunday morning to check it out.

Jacob excitedly began exploring all the knobs and options on it (they were particularly fond of the echo feature), while Oliver wanted to sing. So we found one of his favorite Christmas songs, and here he is singing it with me.

When you have a system with a line in, line out, and several microphone jacks, you can get creative. With a few bits of adapters from my attic, the headset I use for amateur radio worked with it perfectly. Add on a little mic extension cord, and pretty soon Oliver was pretending to be an announcer for a football game!

Then, Oliver decided he would act out a football game while Jacob and I were the announcers.

Something tells me there will be much fun had with this over the next while!

Just wait until I show them how to hook up a handheld radio to it in order to make a remotely-activated loudspeaker…

Share this:

Let me set a scene for you. Two children, ages 7 and 10, are jostling for position. There’s a little pushing and shoving to get the best view.

This is pretty typical for siblings this age. But what, you may wonder, are they trying to see? A TV? Video game?

No. Jacob and Oliver were in a library, trying to see a 98-year-old map of the property owners in Township 23, range 1 East, Harvey County, Kansas. And they were super excited about it, somewhat to the astonishment of the research librarian, who I am sure is more used to children jostling for position over the DVDs in the youth section than poring over maps in the non-circulating historical archives!

Every so often, people stumble upon these abandoned arrows and there is a big discussion online. Even Snopes has had to verify their authenticity (verdict: true). Entire websites exist to tracking and locating the remnants of these arrows. And as one of the early air mail routes went through Kansas, every so often people find these arrows around here.

I got the idea that it would be fun to replicate a journey along the old routes. Maybe I’d spot a few old arrows and such. So I started collecting old maps: a Contract Airmail Route #34 (CAM 34) map from 1927, aviation sectionals from 1933 and 1946, etc.

I noticed an odd thing on these maps: the Newton, KS airport was on the other side of the city from its present location, sometimes even several miles outside the city. What was going on?

(1927 Airway Map)

(1946 Wichita sectional)

So one foggy morning, I explained my puzzlement to the boys. I highlighted all the mysteries: were these maps correct? Were there really two Newton airports at one time? How many airports were there, and where were they? Why did they move? What was the story behind them?

And I offered them the chance to be history detectives with me. And oh my goodness, were they ever excited! We had some information from a very helpful person at the Harvey County Historical Museum (thanks Kris!) So we suspected one airport at least was established in 1927. We also had a description of its location, though given in terms of township maps.

So the boys and I made the short drive over to the museum. We reviewed their property maps, though they were all a little older than the time period we needed. We looked through books and at pictures. Oliver pored over a railroad map of Newton from a century ago, fascinated. Jacob was excited to discover on one map that there used to be a train track down the middle of Main Street! I was interested that the present Newton Airport was once known as Wirt Field, rather to my surprise. I somehow suspect most 2nd and 4th graders spend a lot less excited time on their research floor!

Then on to the Newton Public Library to see if they’d have anything more — and that’s when the map that produced all the excitement came out.

It, by itself, didn’t answer the question, but by piecing together a number of pieces of information — newspaper stories, information from the museum, and the maps — we were able to come up with a pretty good explanation, much to their excitement.

Apparently, a man named Tangeman owned a golf course (the “golf links” according to the paper), and around 1927 the city of Newton purchased it, because of all the planes that were landing there. They turned it into a real airport. Later, they bought land east of the city and moved the airport there. However, during World War II, the Navy took over that location, so they built a third airport a few miles west of the city — but moved back to the current east location after the Navy returned that field to them.

Of course, a project like this just opens up all sorts of extra questions: why isn’t it called Wirt Field anymore? What’s the story of Frank Wirt? What led the Navy to take over Newton’s airport? Why did planes start landing on the golf course? Where precisely was the west airport located? How long was it there? (I found an aerial photo from 1956 that looks like it may have a plane in that general area, but it seems later than I’d have expected)

So now I have the boys interested in going to the courthouse with me to research the property records out there. Jacob is continually astounded that we are discovering things that aren’t in Wikipedia, and also excited that he could be the one to add them. To be continued, apparently!

Share this:

This is morning. Time to fly. Two boys, happy to open the hangar door and get the plane ready.

It’s been a year since I passed the FAA exam and became a pilot. Memories like these are my favorite reminders why I did. It is such fun to see people’s faces light up with the joy of flying a few thousand feet above ground, of the beauty and freedom and peace of the skies.

I’ve flown 14 different passengers in that time; almost every flight I’ve taken has been with people, which I enjoy. I’ve heard “wow” or “beautiful” so many times, and said it myself even more times.

I’ve landed in two state parks, visited any number of wonderful small towns, seen historic sites and placid lakes, ascended magically over forests and plains. I’ve landed at 31 airports in 10 states, flying over 13,000 miles.

Not once have I encountered anyone other than friendly, kind, and outgoing. And why not? After all, we’re working around magic flying carpet machines, right?

(That’s my brother before a flight with me, by the way)

Some weeks it is easy to be glum. This week has been that way for many, myself included. But then, whether you are in the air or on the ground, if you pay attention, you realize we still live in a beautiful world with many wonderful people.

And, in fact, I got a reminder of that this week. Not long after the election, I got in a plane, pushed in the throttle, and started the takeoff roll down a runway in the midst of an Indiana forest. The skies were the best kind of clear blue, and pretty soon I lifted off and could see for miles. Off in the distance, I could see the last cottony remnants of the morning’s fog, lying still in the valleys, surrounding the little farms and houses as if to give them a loving hug. Wow.

Sometimes the flight is bumpy. Sometimes the weather doesn’t cooperate, and it doesn’t happen at all. Sometimes you can fly across four large states and it feels as smooth as glass the whole way.

Whatever happens, at the end of the day, the magic flying carpet machine gets locked up again. We go home, rest our heads on our soft pillows, and if we so choose, remember the beauty we experienced that day.

Really, this post is not about being a pilot. This post is a reminder to pay attention to all that is beautiful in this world. It surrounds us; the smell of pine trees in the forest, the delight in the faces of children, the gentle breeze in our hair, the kind word from a stranger, the very sunrise.

I hope that more of us will pay attention to the moments of clear skies and wind at our back. Even at those moments when we pull the hangar door shut.

When I first heard about the Vintage Computer Festival Midwest, I almost immediately got the notion that I wanted to go. Besides the TRS-80 CoCo II up in my attic, I also have fond memories of an old IBM PC with CGA monitor, a 25MHz 486, an Alpha also in my attic, and a lot of other computers along the way. I didn’t really think my boys would be interested.

But I mentioned it to them, and they just lit up. They remembered the Youtube videos I’d shown them of old line printers and punch card readers, and thought it would be great fun. I thought it could be a great educational experience for them too — and it was.

It also turned into a trip that combined being a proud dad with so many of my other interests. Quite a fun time.

(Jacob modeling his new t-shirt)

Captain Jacob

Chicago being not all that close to Kansas, I planned to fly us there. If you’re flying yourself, solid flight planning is always important. I had already planned out my flight using electronic tools, but I always carry paper maps with me in the cockpit for backup. I got them out and the boys and I planned out the flight the old-fashioned way.

Here’s Oliver using a scale ruler (with markings for miles corresponding to the scale of the map) and Jacob doing calculating for us. We measured the entire route and came to within one mile of the computer’s calculation for each segment — those boys are precise!

We figured out how much fuel we’d use, where we’d make fuel stops, etc.

The day of our flight, we made it as far as Davenport, Iowa when a chance of bad weather en route to Chicago convinced me to land there and drive the rest of the way. The boys saw that as part of the exciting adventure!

Jacob is always interested in maps, and had kept wanting to use my map whenever we flew. So I dug an old Android tablet out of the attic, put Avare on it (which has aviation maps), and let him use that. He was always checking it while flying, sometimes saying this over his headset: “DING. Attention all passengers, this is Captain Jacob speaking. We are now 45 miles from St. Joseph. Our altitude is 6514 feet. Our speed is 115 knots. We will be on the ground shortly. Thank you. DING”

Here he is at the Davenport airport, still busy looking at his maps:

Every little airport we stopped at featured adults smiling at the boys. People enjoyed watching a dad and his kids flying somewhere together.

Oliver kept busy too. He loves to help me on my pre-flight inspections. He will report every little thing to me – a scratch, a fleck of paint missing on a wheel cover, etc. He takes it seriously. Both boys love to help get the plane ready or put it away.

The Computers

Jacob quickly gravitated towards a few interesting things. He sat for about half an hour watching this old Commodore plotter do its thing (click for video):

His other favorite thing was the phones. Several people had brought complete analog PBXs with them. They used them to demonstrate various old phone-related hardware; one had several BBSs running with actual modems, another had old answering machines and home-security devices. Jacob learned a lot about phones, including how to operate a rotary-dial phone, which he’d never used before!

Oliver was drawn more to the old computers. He was fascinated by the IBM PC XT, which I explained was just about like a model I used to get to use sometimes. They learned about floppy disks and how computers store information.

He hadn’t used joysticks much, and found Pong (“this is a soccer game!”) interesting. Somebody has also replaced the guts of a TRS-80 with a Raspberry Pi running a SNES emulator. This had thoroughly confused me for a little while, and excited Oliver.

Jacob enjoyed an old TRS-80, which, through a modern Ethernet interface and a little computation help in AWS, provided an interface to Wikipedia. Jacob figured out the text-mode interface quickly. Here he is reading up on trains.

I had no idea that Commodore made a lot of adding machines and calculators before they got into the home computer business. There was a vast table with that older Commodore hardware, too much to get on a single photo. But some of the adding machines had their covers off, so the boys got to see all the little gears and wheels and learn how an adding machine can do its printing.

And then we get to my favorite: the big iron. Here is a VAX — a working VAX. When you have a computer that huge, it’s easier for the kids to understand just what something is.

When we encountered the table from the Glenside Color Computer Club, featuring the good old CoCo IIs like what I used as a kid (and have up in my attic), I pointed out to the boys that “we have a computer just like this that can do these things” — and they responded “wow!” I think they are eager to try out floppy disks and disk BASIC now.

Some of my favorites were the old Unix systems, which are a direct ancestor to what I’ve been working with for decades now. Here’s AT&T System V release 3 running on its original hardware:

And there were a couple of Sun workstations there, making me nostalgic for my college days. If memory serves, this one is actually running on m68k in the pre-Sparc days:

Returning home

After all the excitement of the weekend, both boys zonked out for awhile on the flight back home. Here’s Jacob, sleeping with his maps still up.

As we were nearly home, we hit a pocket of turbulence, the kind that feels as if the plane is dropping a bit (it’s perfectly normal and safe; you’ve probably felt that on commercial flights too). I was a bit concerned about Oliver; he is known to get motion sick in cars (and even planes sometimes). But what did I hear from Oliver?

“Whee! That was fun! It felt like a roller coaster! Do it again, dad!”