The Changeloghttp://changelog.complete.org
Comments on family, technology, and societyMon, 06 Nov 2017 01:44:12 +0000en-UShourly1https://wordpress.org/?v=4.8.15552872The Yellow House Phone Company (Featuring Asterisk and an 11-year-old)http://changelog.complete.org/archives/9818-the-yellow-house-phone-company-featuring-asterisk-and-an-11-year-old
http://changelog.complete.org/archives/9818-the-yellow-house-phone-company-featuring-asterisk-and-an-11-year-old#respondMon, 06 Nov 2017 01:44:12 +0000http://changelog.complete.org/?p=9818Continue reading The Yellow House Phone Company (Featuring Asterisk and an 11-year-old)→]]>“Well Jacob, do you think we should set up our own pretend phone company in the house?”

“We can DO THAT?”

“Yes!”

“Then… yes. Yes! YES YES YESYESYESYES YES! Let’s do it, dad!”

Not long ago, my parents had dug up the old phone I used back in the day. We still have a landline, and Jacob was having fun discovering how an analog phone works. I told him about the special number he could call to get the time and temperature read out to him. He discovered what happens if you call your own number and hang up. He figured out how to play “Mary Had a Little Lamb” using touchtone keys (after a slightly concerned lecture from me setting out some rules to make sure his “musical dialing” wouldn’t result in any, well, dialing.)

He was hooked. So I thought that taking it to the next level would be a good thing for a rainy day. I have run Asterisk before, though I had unfortunately gotten rid of most of my equipment some time back. But I found a great deal on a Cisco 186 ATA (Analog Telephone Adapter). It has two FXS lines (FXS ports simulate the phone company, and provide dialtone and ring voltage to a connected phone), and of course hooks up to the LAN.

We plugged that in, and Jacob was amazed to see its web interface come up. I had to figure out how to configure it (unfortunately, it uses SCCP rather than SIP, and figuring out Asterisk’s chan_skinny took some doing, but we got there.)

I set up voicemail. He loved it. He promptly figured out how to record his own greetings. We set up a second phone on the other line, so he could call between them. The cordless phones in our house support SIP, so I configured one of them as a third line. He spent a long time leaving himself messages.

Pretty soon we both started having ideas. I set up extension 777, where he could call for the time. Then he wanted a way to get the weather forecast. Well, weather-util generates a text-based report. With it, a little sed and grep tweaking, the espeak TTS engine, and a little help from sox, I had a shell script worked up that would read back a forecast whenever he called a certain extension. He was super excited! “That’s great, dad! Can it also read weather alerts too?” Sure! weather-util has a nice option just for that. Both boys cackled as the system tried to read out the NWS header (their timestamps like 201711031258 started with “two hundred one billion…”)

Then I found an online source for streaming NOAA Weather Radio feeds – Jacob enjoys listening to weather radio – and I set up another extension he could call to listen to that. More delight!

But it really took off when I asked him, “Would you like to record your own menu?” “You mean those things where it says press 1 or 2 for this or that?” “Yes.” “WE CAN DO THAT?” “Oh yes!” “YES, LET’S DO IT RIGHT NOW!”

So he recorded a menu, then came and hovered by me while I hacked up extensions.conf, then eagerly went back to the phone to try it. Oh the excitement of hearing hisown voice, and finding that it worked! Pretty soon he was designing sub-menus (“OK Dad, so we’ll set it up so people can press 2 for the weather, and then choose if they want weather radio or the weather report. I’m recording that now. Got it?”)

He has informed me that next Saturday we will build an intercom system “like we have at school.” I’m going to have to have some ideas on how to tie Squeezebox in with Asterisk to make that happen, I think. Maybe this will do.

]]>http://changelog.complete.org/archives/9818-the-yellow-house-phone-company-featuring-asterisk-and-an-11-year-old/feed09818Switching to xmonad + Gnome – and ditching a Machttp://changelog.complete.org/archives/9815-switching-to-xmonad-gnome-and-ditching-a-mac
http://changelog.complete.org/archives/9815-switching-to-xmonad-gnome-and-ditching-a-mac#commentsThu, 07 Sep 2017 02:43:12 +0000http://changelog.complete.org/?p=9815Continue reading Switching to xmonad + Gnome – and ditching a Mac→]]>I have been using XFCE with xmonad for years now. I’m not sure exactly how many, but at least 6 years, if not closer to 10. Today I threw in the towel and switched to Gnome.

More recently, at a new job, I was given a Macbook Pro. I wasn’t entirely sure what to think of this, but I thought I’d give it a try. I found MacOS to be extremely frustrating and confining. It had no real support for a tiling window manager, and although projects like amethyst tried to approximate what xmonad can do on Linux, they were just too limited by the platform and were clunky. Moreover, the entire UI was surprisingly sluggish; maybe that was an induced effect from animations, but I don’t think that explains it. A Debisn stretch install, even on inferior hardware, was snappy in a way that MacOS never was. So I have requested to swap for a laptop that will run Debian. The strange use of Command instead of Control for things, combined with the overall lack of configurability of keybindings, meant that I was going to always be fighting muscle memory moving from one platform to another. Not only that, but being back in the world of a Free Software OS means a lot.

Now then, back to xmonad and XFCE situation. XFCE once worked very well with xmonad. Over the years, this got more challenging. Around the jessie (XFCE 4.10) time, I had to be very careful about when I would let it save my session, because it would easily break. With stretch, I had to write custom scripts because the panel wouldn’t show up properly, and even some application icons would be invisible, if things were started in a certain order. This took much trial and error and was still cumbersome.

Gnome 3, with its tightly-coupled Gnome Shell, has never been compatible with other window managers — at least not directly. A person could have always used MATE with xmonad — but a lot of people that run XFCE tend to have some Gnome 3 apps (for instance, evince) anyhow. Cinnamon also wouldn’t work with xmonad, because it is simply another tightly-coupled shell instead of Gnome Shell. And then today I discovered gnome-flashback. gnome-flashback is a Gnome 3 environment that uses the traditional X approach with a separate window manager (metacity of yore by default). Sweet.

It turns out that Debian’s xmonad has built-in support for it. If you know the secret: apt-get install gnome-session-flashback (OK, it’s not so secret; it’s even in xmonad’s README.Debian these days) Install that, plus gnome and gdm3 and things are nice. Configure xmonad with GNOME support and poof – goodness right out of the box, selectable from the gdm sessions list.

I still have some gripes about Gnome’s configurability (or lack thereof). But I’ve got to say: This environment is the first one I’ve ever used that got external display switching very nearly right without any configuration, and I include MacOS in that. Plug in an external display, and poof – it’s configured and set up. You can hit a toggle key (Windows+P by default) to change the configurations, or use the Display section in gnome-control-center. Unplug it, and it instantly reconfigures itself to put everything back on the laptop screen. Yessss! I used to have scripts to do this in the wheezy/jessie days. XFCE in stretch had numerous annoying failures in this area which rendered the internal display completely dark until the next reboot – very frustrating. With Gnome, it just works. And, even if you have “suspend on lid closed” turned on, if the system is powered up and hooked up to an external display, it will keep running even if the lid is closed, figuring you must be using it on the external screen. Another thing the Mac wouldn’t do well.

All in all, some pretty good stuff here. I continue to be impressed by stretch. It is darn impressive to put this OS on generic hardware and have it outshine the closed-ecosystem Mac!

]]>http://changelog.complete.org/archives/9815-switching-to-xmonad-gnome-and-ditching-a-mac/feed39815The Joy of Exploring: Old Phone Systems, Pizza, and Discoveryhttp://changelog.complete.org/archives/9811-the-joy-of-exploring-old-phone-systems-pizza-and-discovery
http://changelog.complete.org/archives/9811-the-joy-of-exploring-old-phone-systems-pizza-and-discovery#commentsMon, 28 Aug 2017 01:54:34 +0000http://changelog.complete.org/?p=9811Continue reading The Joy of Exploring: Old Phone Systems, Pizza, and Discovery→]]>This story involves boys pretending to be pizza deliverymen using a working automated Strowger telephone exchange demonstrator on display in a museum, which is very old and is, to my knowledge, the only such working exhibit in the world. (Yes, I have video.) But first, a thought on exploration.

There are those that would say that there is nothing left to explore anymore – that the whole earth is mapped, photographed by satellites, and, well, known.

I prefer to look at it a different way: the earth is full of places that billions of people will never see, and probably don’t even know about. Those places may be quiet country creeks, peaceful neighborhoods one block away from major tourist attractions, an MTA museum in Brooklyn, a state park in Arkansas, or a beautiful church in Germany.

Martha is not yet two months old, and last week she and I spent a surprisingly long amount of time just gazing at tree branches — she was mesmerized, and why not, because to her, everything is new.

As I was exploring in Portland two weeks ago, I happened to pick up a nearly-forgotten book by a nearly-forgotten person, Beryl Markham, a woman who was a pilot in Africa about 80 years ago. The passage that I happened to randomly flip to in the bookstore, which really grabbed my attention, was this:

The available aviation maps of Africa in use at that time all bore the cartographer’s scale mark, ‘1/2,000,000’ — one over two million. An inch on the map was about thitry-two miles in the air, as compared to the flying maps of Europe on which one inch represented no more than four air miles.

Moreover, it seemed that the printers of the African maps had a slightly malicious habit of including, in large letters, the names of towns, junctions, and villages which, while most of them did exist in fact, as a group of thatched huts may exist or a water hold, they were usually so inconsequential as completely to escape discovery from the cockpit.

Beyond this, it was even more disconcerting to examine your charts before a proposed flight only to find that in many cases the bulk of the terrain over which you had to fly was bluntly marked: ‘UNSURVEYED’.

It was as if the mapmakers had said, “We are aware that between this spot and that one, there are several hundred thousands of acres, but until you make a forced landing there, we won’t know whether it is mud, desert, or jungle — and the chances are we won’t know then!”

— Beryl Markham, West With the Night

My aviation maps today have no such markings. The continent is covered with radio beacons, the world with GPS, the maps with precise elevations of the ground and everything from skyscrapers to antenna towers.

And yet, despite all we know, the world is still a breathtaking adventure.

Yesterday, the boys and I were going to fly to Abilene, KS, to see a museum (Seelye Mansion). Circumstances were such that we neither flew, nor saw that museum. But we still went to Abilene, and wound up at the Museum of Independent Telephony, a wondrous place for anyone interested in the history of technology. As it is one of those off-the-beaten-path sorts of places, the boys got 2.5 hours to use the hands-on exhibits of real old phones, switchboards, and also the schoolhouse out back. They decided — why not? — to use this historic equipment to pretend to order pizzas.

Jacob and Oliver proceeded to invent all sorts of things to use the phones for: ordering pizza, calling the cops to chase the pizza delivery guys, etc. They were so interested that by 2PM we still hadn’t had lunch and they claimed “we’re not hungry” despite the fact that we were going to get pizza for lunch. And I certainly enjoyed the exhibits on the evolution of telephones, switching (from manual plugboards to automated switchboards), and such.

This place was known – it even has a website, I had been there before, and in fact so had the boys (my parents took them there a couple of years ago). But yesterday, we discovered the Strowger switch had been repaired since the last visit, and that it, in fact, is great for conversations about pizza.

Whether it’s seeing an eclipse, discovering a fascination with tree branches, or historic telephones, a spirit of curiosity and exploration lets a person find fun adventures almost anywhere.

]]>http://changelog.complete.org/archives/9811-the-joy-of-exploring-old-phone-systems-pizza-and-discovery/feed29811The Eclipsehttp://changelog.complete.org/archives/9808-the-eclipse
http://changelog.complete.org/archives/9808-the-eclipse#respondTue, 22 Aug 2017 15:17:28 +0000http://changelog.complete.org/?p=9808Continue reading The Eclipse→]]>Highway US-81 in northern Kansas and southern Nebraska is normally a pleasant, sleepy sort of drive. It was upgraded to a 4-lane road not too long ago, but as far as 4-lane roads go, its traffic is typically light. For drives from Kansas to South Dakota, it makes a pleasant route.

Yesterday was eclipse day. I strongly suspect that highway 81 had more traffic that day than it ever has before, or ever will again. For nearly the entire 3-hour drive to Geneva, NE, it was packed — though mostly still moving at a good speed. And for our entire drive back, highway 81 and every other southbound road we used was so full it felt like rush hour in Dallas. (Well, not quite. Traffic was still moving.) I believe scenes like this were played out across the continent.

I’ve been taking a lot of photos, and writing about our new baby Martha lately. Now it’s time to write a bit about some more adventures with Jacob and Oliver – they’re now in third and fifth grades in school.

We had been planning to fly, and airports I called were either full, or were planning to park planes in the grass, or even shut down some runways to use for parking. The airport in the little town of Beatrice, NE (which I had visited twice before) was even going to have a temporary FAA control tower. At the last minute, due to some storm activity near home at departure time, we unloaded the plane and drove instead.

The atmosphere at the fairgrounds in Geneva was festive. One family had brought bubbles for their kids — and extras to share.

I had bought the boys a book about the eclipse, which they were reading before and during the event. They were both great, safe users of their eclipse glasses.

Jacob caught a toad, and played with it for awhile. He wanted to bring it home with us, but I convinced him to let me take a picture of him with his toad friend instead.

While we were waiting for totality, a number of buses from the local school district arrived. So by the time the big moment arrived, we could hear the distant roar of delight and applause from the school children gathered at the far end of the field, plus all the excitement nearby. Both boys were absolutely ecstatic to be witnessing it (and so was I!) “Wow!” “Awesome!” And simple cackles of delight were heard. On the drive home, they both kept talking about how amazing it was, and it was “once in a lifetime.”

We enjoyed our “eclipse neighbors” – the woman from San Antonio next to us, the surprise discovery of another family from just a few miles from us parked two cars down, even running into relatives at a restaurant on the way home. The applause from all around when it started – and when it ended. And the feeling, which is hard to describe, of awe and amazement at the wonders of our world and our universe.

There are many problems with the world right now, but somehow there’s something right about people coming together from all over to enjoy it.

A month ago, we were waiting for our new baby; time seemed to stand still. Now she is here! Martha Goerzen was born recently, and she is doing well and growing! Laura and I have enjoyed moments of cuddling her, watching her stare at our faces, hearing her (hopefully) soft sounds as she falls asleep in our arms. It is also heart-warming to see Martha’s older brothers take such an interest in her. Here is the first time Jacob got to hold her:

Oliver, who is a boy very much into sports, play involving police and firefighters, and such, has started adding “aww” and “she’s so cute!” to his common vocabulary. He can be very insistent about interrupting me to hold her, too.

]]>http://changelog.complete.org/archives/9805-a-new-baby-and-deep-smiles/feed19805Time, Frozenhttp://changelog.complete.org/archives/9799-time-frozen
http://changelog.complete.org/archives/9799-time-frozen#commentsTue, 04 Jul 2017 03:00:30 +0000http://changelog.complete.org/?p=9799Continue reading Time, Frozen→]]>We’re expecting a baby any time now. The last few days have had an odd quality of expectation: any time, our family will grow.

It makes time seem to freeze, to stand still.

We have Jacob, about to start fifth grade and middle school. But here he is, still a sweet and affectionate kid as ever. He loves to care for cats and seeks them out often. He still keeps an eye out for the stuffed butterfly he’s had since he was an infant, and will sometimes carry it and a favorite blanket around the house. He will also many days prepare the “Yellow House News” on his computer, with headlines about his day and some comics pasted in — before disappearing to play with Legos for awhile.

And Oliver, who will walk up to Laura and “give baby a hug” many times throughout the day — and sneak up to me, try to touch my arm, and say “doink” before running off before I can “doink” him back. It was Oliver that had asked for a baby sister for Christmas — before he knew he’d be getting one!

In the past week, we’ve had out the garden hose a couple of times. Both boys will enjoy sending mud down our slide, or getting out the “water slide” to play with, or just playing in mud. The rings of dirt in the bathtub testify to the fun that they had. One evening, I built a fire, we made brats and hot dogs, and then Laura and I sat visiting and watching their water antics for an hour after, laughter and cackles of delight filling the air, and cats resting on our laps.

These moments, or countless others like Oliver’s baseball games, flying the boys to a festival in Winfield, or their cuddles at bedtime, warm the heart. I remember their younger days too, with fond memories of taking them camping or building a computer with them. Sometimes a part of me wants to just keep soaking in things just as they are; being a parent means both taking pride in children’s accomplishments as they grow up, and sometimes also missing the quiet little voice that can be immensely excited by a caterpillar.

And yet, all four of us are so excited and eager to welcome a new life into our home. We are ready. I can’t wait to hold the baby, or to lay her to sleep, to see her loving and excited older brothers. We hope for a smooth birth, for mom and baby.

Here is the crib, ready, complete with a mobile with a cute bear (and even a plane). I can’t wait until there is a little person here to enjoy it.

]]>http://changelog.complete.org/archives/9799-time-frozen/feed19799First Experiences with Stretchhttp://changelog.complete.org/archives/9797-first-experiences-with-stretch
http://changelog.complete.org/archives/9797-first-experiences-with-stretch#commentsThu, 22 Jun 2017 13:19:37 +0000http://changelog.complete.org/?p=9797Continue reading First Experiences with Stretch→]]>I’ve done my first upgrades to Debian stretch at this point. The results have been overall good. On the laptop my kids use, I helped my 10-year-old do it, and it worked flawlessly. On my workstation, I got a kernel panic on boot. Hmm.

Unfortunately, my system has to use the nv drivers, which leaves me with an 80×25 text console. It took some finagling (break=init in grub, then manually insmoding the appropriate stuff based on modules.dep for nouveau), but finally I got a console so I could see what was breaking. It appeared that init was crashing because it couldn’t find liblz4. A little digging shows that liblz4 is in /usr, and /usr wasn’t mounted. I’ve filed the bug on systemd-sysv for this.

I run root on ZFS, and further digging revealed that I had datasets named like this:

tank/hostname-1/ROOT

tank/hostname-1/usr

tank/hostname-1/var

This used to be fine. The mountpoint property of the usr dataset put it at /usr without incident. But it turns out that this won’t work now, unless I set ZFS_INITRD_ADDITIONAL_DATASETS in /etc/default/zfs for some reason. So I renamed them so usr was under ROOT, and then the system booted.

Then I ran samba not liking something in my bind interfaces line (to be fair, it did still say eth0 instead of br0). rpcbind was failing in postinst, though a reboot seems to have helped that. More annoying was that I had trouble logging into my system because resolv.conf was left empty (despite dns-* entries in /etc/network/interfaces and the presence of resolvconf). I eventually repaired that, and found that it kept removing my “search” line. Eventually I removed resolvconf.

Then mariadb’s postinst was silently failing. I eventually discovered it was sending info to syslog (odd), and /etc/init.d/apparmor teardown let it complete properly. It seems like there may have been an outdated /etc/apparmor.d/cache/usr.sbin.mysql out there for some reason.

Then there was XFCE. I use it with xmonad, and the session startup was really wonky. I had to zap my sessions, my panel config, etc. and start anew. I am still not entirely sure I have it right, but I at do have a usable system now.

]]>http://changelog.complete.org/archives/9797-first-experiences-with-stretch/feed59797Fixing the Problems with Docker Imageshttp://changelog.complete.org/archives/9794-fixing-the-problems-with-docker-images
http://changelog.complete.org/archives/9794-fixing-the-problems-with-docker-images#commentsFri, 09 Jun 2017 17:58:46 +0000http://changelog.complete.org/?p=9794Continue reading Fixing the Problems with Docker Images→]]>I recently wrote about the challenges in securing Docker container contents, and in particular with keeping up-to-date with security patches from all over the Internet.

Today I want to fix that.

Besides security, there is a second problem: the common way of running things in Docker pretends to provide a traditional POSIX API and environment, but really doesn’t. This is a big deal.

Before diving into that, I want to explain something: I have often heard it said the Docker provides single-process containers. This is unambiguously false in almost every case. Any time you have a shell script inside Docker that calls cp or even ls, you are running a second process. Web servers from Apache to whatever else use processes or threads of various types to service multiple connections at once. Many Docker containers are single-application, but a process is a core part of the POSIX API, and very little software would work if it was limited to a single process. So this is my little plea for more precise language. OK, soapbox mode off.

Now then, in a traditional Linux environment, besides your application, there are other key components of the system. These are usually missing in Docker containers.

So today, I will fix this also.

In my docker-debian-base images, I have prepared a system that still has only 11MB RAM overhead, makes minimal changes on top of Debian, and yet provides a very complete environment and API. Here’s what you get:

Working syslog, which can either export all logs to Docker’s logging infrastructure, or keep them within the container, depending on your preferences.

Working real schedulers (cron, anacron, and at), plus at least the standard logrotate utility to help prevent log files inside the container from becoming huge.

The above goes into my “minimal” image. Additional images add layers on top of it, and here are some of the features they add:

A real SMTP agent (exim4-daemon-light) so that cron and friends can actually send you mail

SSH client and server (optionally exposed to the Internet)

Automatic security patching via unattended-upgrades and needsrestart

All of the above, including the optional features, has an 11MB overhead on start. Not bad for so much, right?

From here, you can layer on top all your usual Dockery things. You can still run one application per container. But you can now make sure your disk doesn’t fill up from logs, run your database vacuuming commands at will, have your blog download its RSS feeds every few minutes, etc — all from within the container, as it should be. Furthermore, you don’t have to reinvent the wheel, because Debian already ships with things to take care of a lot of this out of the box — and now those tools will just work.

By using sysvinit rather than runit, the OS default init scripts can be used unmodified, reducing the administrative burden on container builders

Phusion’s system is, for some reason, not auto-built on the Docker hub. Mine is, so it will be automatically revised whenever the underlying Debian system, or the Github repository, is.

Finally a word on the choice to use sysvinit. It would have been simpler to use systemd here, since it is the default in Debian these days. Unfortunately, systemd requires you to poke some holes in the Docker security model, as well as mount a cgroups filesystem from the host. I didn’t consider this acceptable, and sysvinit ran without these workarounds, so I went with it.

With all this, Docker becomes a viable replacement for KVM for various services on my internal networks. I’ll be writing about that later.

]]>http://changelog.complete.org/archives/9794-fixing-the-problems-with-docker-images/feed59794Family Spring: A Story in Photoshttp://changelog.complete.org/archives/9792-family-spring-a-story-in-photos
http://changelog.complete.org/archives/9792-family-spring-a-story-in-photos#commentsTue, 06 Jun 2017 02:32:45 +0000http://changelog.complete.org/?p=9792Continue reading Family Spring: A Story in Photos→]]>This has been a spring with times to relax, times to be busy, times of anticipation of a new baby, and times of enjoying our family.

Rather than write a lot of words about it, I’m telling the story in photos.

To view, click here, then click Show Info in the upper right to see captions. You can pause it with the button in the lower left, and use arrow keys to advance.

]]>http://changelog.complete.org/archives/9792-family-spring-a-story-in-photos/feed29792Flying with my brothershttp://changelog.complete.org/archives/9789-flying-with-my-brothers
http://changelog.complete.org/archives/9789-flying-with-my-brothers#respondMon, 05 Jun 2017 01:36:45 +0000http://changelog.complete.org/?p=9789Continue reading Flying with my brothers→]]>Picture one Sunday morning. Three guys are seemingly-randomly walking into a Mennonite church in rural Nebraska. One with long hair and well-maintained clothes from the 70s. Another dressed well enough to be preaching. And the third simply dressed to be comfortable, with short hair showing evidence of having worn a headset for a couple of hours that morning. This was the scene as we made a spur-of-the-moment visit to that church — which resulted in quite some surprise all around, since my brother knew a number of people there.

For instance:

Pastor: Peter! What are you doing here?

Peter: [jokingly] Is that how you greet visitors here?

And then, of course, Peter would say, “Well, we were flying home from South Dakota and figured we’d stop in at Beatrice for fuel. And drop in on you.” Followed by some surprise that we would stop at their little airport (which is quite a nice one).

This all happened because it was windy. This is the fun adventure of aviation. Sometimes you plan to go to Texas, but the weather there is terrible, so you discover a 100-year-old landmark in Indiana instead. Or sometimes, like a couple of weeks ago, we planned to fly straight home but spent a few hours exploring rural Nebraska.

The three of us flew to Sioux Falls, SD, in a little Cessna to visit my uncle and aunt up there. On our flight up, we stopped at the little airport in Seward, NE. It was complete with this unique elevated deck. In my imagination, this is used for people to drink beer while watching the planes land.

In South Dakota, we had a weekend full of card and board games, horseshoes, and Crokinole with my uncle and aunt, who are always fun to visit. We had many memories of visits up there as children — and the pleasant enjoyment of the fact that we didn’t need an 8-hour drive to get there. We flew back with a huge bag of large rhubarb from their garden (that too is something of a tradition!)

It was a fun weekend to spend with my brothers — first time we’d been able to do this in a long while. And it marked the 11th state I’ve flown into, and over 17,000 miles of flying.

]]>http://changelog.complete.org/archives/9789-flying-with-my-brothers/feed09789Is there any way to truly secure Docker container contents?http://changelog.complete.org/archives/9782-is-there-any-way-to-truly-secure-docker-container-contents
http://changelog.complete.org/archives/9782-is-there-any-way-to-truly-secure-docker-container-contents#commentsSat, 29 Apr 2017 18:41:30 +0000http://changelog.complete.org/?p=9782Continue reading Is there any way to truly secure Docker container contents?→]]>There is much to like about Docker. Much has been written about it, and about how secure the containerization is.

This post isn’t about that. This is about keeping what’s inside each container secure. I believe we have a fundamental problem here.

Let’s say you want to run a PHP application like WordPress under Apache. Here are the things you need to keep secure:

WordPress itself

All plugins, themes, customizations

All PHP libraries it uses (MySQL, image-processing, etc.)

MySQL

Apache

All libraries MySQL or Apache use: OpenSSL, libc, PHP itself, etc.

The kernel

All containerization tools

On Debian (and most of its best-known derivatives), we are extremely lucky to have a wonderful security support system. If you run a Debian system, the combination of unattended-updates, needrestart, debsecan, and debian-security-support will help one keep a Debian system secure and verify it is. When the latest OpenSSL bug comes out, generally speaking by the time I wake up, unattended-updates has already patched it, needrestart has already restarted any server that uses it, and I’m protected. Debian’s security team generally backports fixes rather than just say “here’s the new version”, making it very safe to automatically apply patches. As long as I use what’s in Debian stable, all layers mentioned above will be protected using this scheme.

This picture is much nicer than what we see in Docker.

Problems

We have a lot of problems in the Docker ecosystem:

No built-in way to know when a base needs to be updated, or to automatically update it

Diverse and complicated vendor security picture

No way to detect when intermediate libraries need to be updated

Complicated final application security picture

Let’s look at them individually.

Problem #1: No built-in way to know when a base needs to be updated, or to automatically update it

First of all, there is nothing in Docker like unattended-updates. Although a few people have suggested ways to run unattended-updates inside containers, there are many reasons that approach doesn’t work well. The standard advice is to update/rebuild containers.

So how do you know when to do that? It is not all that obvious. Theoretically, official OS base images will be updated when needed, and then other Docker hub images will detect the base update and be rebuilt. So, if a bug in a base image is found, and if the vendors work properly, and if you are somehow watching, then you could be protected. There is work in this area; tools such as watchtower help here.

But this can lead to a false sense of security, because:

Problem #2: Diverse and complicated vendor security picture

Different images can use different operating system bases. Consider just these official images, and the bases they use: (tracking latest tag on each)

nginx: debian:stretch-slim (stretch is pre-release at this date!)

mysql: debian:jessie

mongo: debian:wheezy-slim (previous release)

apache httpd: debian:jessie-backports

postgres: debian:jessie

node: buildpack-deps:jessie, eventually depends on debian:jessie

wordpress: php:5.6-apache, eventually depends on debian:jessie

And how about a few unofficial images?

oracle/openjdk: oraclelinux:latest

robotamer/citadel: debian:testing (dangerous, because testing is an alias for different distros at different times)

docker.elastic.co/kibana: ubuntu of some sort

The good news is that Debian jessie seems to be pretty popular here. The bad news is that you see everything from Oracle Linux, to Ubuntu, to Debian testing, to Debian oldstable in just this list. Go a little further, and you’ll see Alpine Linux, CentOS, and many more represented.

Here’s the question: what do you know about the security practices of each of these organizations? How well updated are their base images? Even if it’s Debian, how well updated is, for instance, the oldstable or the testing image?

The attack surface here is a lot larger than if you were just using a single OS. But wait, it gets worse:

Problem #3: No way to detect when intermediate libraries need to be updated

Let’s say your Docker image is using a base that is updated immediately when a security problem is found. Let’s further assume that your software package (WordPress, MySQL, whatever) is also being updated.

What about the intermediate dependencies? Let’s look at the build process for nginx. The Dockerfile for it begins with Debian:stretch-slim. But then it does a natural thing: it runs an apt-get install, pulling in packages from both Debian and an nginx repo.

I ran the docker build across this. Of course, the apt-get command brings in not just the specified packages, but also their dependencies. Here are the ones nginx brought in:

Now, what is going to trigger a rebuild if there’s a security fix to libssl1.1 or libicu57? (Both of these have a history of security holes.) The answer, for the vast majority of Docker images, seems to be: nothing automatic.

Problem #4: Complicated final application security picture

And that brings us to the last problem: Let’s say you want to run an application in Docker. exim, PostgreSQL, Drupal, or maybe something more obscure. Who is watching for security holes in it? If you’re using Debian packages, the Debian security team is. If you’re using a Docker image, well, maybe it’s the random person that contributed it, maybe it’s the vendor, maybe it’s Docker, maybe it’s nobody. You have to take this burden on yourself, to validate the security support picture for each image you use.

Conclusion

All this adds up to a lot of work, which is not taken care of for you by default in Docker. It is no surprise that many Docker images are insecure, given this picture. The unfortunate reality is that many Docker containers are running with known vulnerabilities that have known fixes, but just aren’t, and that’s sad.

I wonder if there are any practices people are using that can mitigate this better than what the current best-practice recommendations seem to be?

]]>http://changelog.complete.org/archives/9782-is-there-any-way-to-truly-secure-docker-container-contents/feed69782Parsing the GOP’s Health Insurance Statisticshttp://changelog.complete.org/archives/9773-parsing-the-gops-health-insurance-statistics
http://changelog.complete.org/archives/9773-parsing-the-gops-health-insurance-statistics#respondTue, 14 Mar 2017 15:35:05 +0000http://changelog.complete.org/?p=9773Continue reading Parsing the GOP’s Health Insurance Statistics→]]>There has been a lot of noise lately about the GOP health care plan (AHCA) and the differences to the current plan (ACA or Obamacare). A lot of statistics are being misinterpreted.

The New York Times has an excellent analysis of some of this. But to pick it apart, I want to highlight a few things:

Many Republicans are touting the CBO’s estimate that, some years out, premiums will be 10% lower under their plan than under the ACA. However, this carries with it a lot of misleading information.

First of all, many are spinning this as if costs would go down. That’s not the case. The premiums would still rise — they would just have risen less by the end of the period than under ACA. That also ignores the immediate spike and throwing millions out of the insurance marketplace altogether.

Now then, where does this 10% number come from? First of all, you have to understand the older people are substantially more expensive to the health system, and therefore more expensive to insure. ACA limited the price differential from the youngest to the oldest people, which meant that in effect some young people were subsidizing older ones on the individual market. The GOP plan removes that limit. Combined with other changes in subsidies and tax credits, this dramatically increases the cost to older people. For instance, the New York Times article cites a CBO estimate that “the price an average 64-year-old earning $26,500 would need to pay after using a subsidy would increase from $1,700 under Obamacare to $14,600 under the Republican plan.”

They further conclude that these exceptionally high rates would be so unaffordable to older people that the older people will simply stop buying insurance on the individual market. This means that the overall risk pool of people in that market is healthier, and therefore the average price is lower.

So, to sum up: the reason that insurance premiums under the GOP plan will rise at a slightly slower rate long-term is that the higher-risk people will be unable to afford insurance in the first place, leaving only the cheaper people to buy in.

This means there was a data error on the drive. But it’s worse than a typical data error — this is an error that was not detected by the hardware. Unlike most filesystems, ZFS and btrfs write a checksum with every block of data (both data and metadata) written to the drive, and the checksum is verified at read time. Most filesystems don’t do this, because theoretically the hardware should detect all errors. But in practice, it doesn’t always, which can lead to silent data corruption. That’s why I use ZFS wherever I possibly can.

As I looked into this issue, I saw that ZFS repaired about 400KB of data. I thought, “well, that was unlucky” and just ignored it.

Then a week later, it happened again. Pretty soon, I noticed it happened every Sunday, and always to the same drive in my pool. It so happens that the highest I/O load on the machine happens on Sundays, because I have a cron job that runs zpool scrub on Sundays. This operation forces ZFS to read and verify the checksums on every block of data on the drive, and is a nice way to guard against unreadable sectors in rarely-used data.

I finally swapped out the drive, but to my frustration, the new drive now exhibited the same issue. The SATA protocol does include a CRC32 checksum, so it seemed (to me, at least) that the problem was unlikely to be a cable or chassis issue. I suspected motherboard.

It so happened I had a 9211-8i SAS card. I had purchased it off eBay awhile back when I built the server, but could never get it to see the drives. I wound up not filling it up with as many drives as planned, so the on-board SATA did the trick. Until now.

As I poked at the 9211-8i, noticing that even its configuration utility didn’t see any devices, I finally started wondering if the SAS/SATA breakout cables were a problem. And sure enough – I realized I had a “reverse” cable and needed a “forward” one. $14 later, I had the correct cable and things are working properly now.

One other note: RAM errors can sometimes cause issues like this, but this system uses ECC DRAM and the errors would be unlikely to always manifest themselves on a particular drive.

So over the course of this, had I not been using ZFS, I would have had several megabytes of reads with undetected errors. Thanks to using ZFS, I know my data integrity is still good.

]]>http://changelog.complete.org/archives/9769-silent-data-corruption-is-real/feed149769What is happening to America?http://changelog.complete.org/archives/9762-what-is-happening-to-america
http://changelog.complete.org/archives/9762-what-is-happening-to-america#commentsThu, 26 Jan 2017 04:17:54 +0000http://changelog.complete.org/?p=9762Continue reading What is happening to America?→]]>I still remember vividly my first visit to Europe, back in 2010. I had just barely gotten off a plane in Hamburg and on to a bus to Lubeck, and struck up a conversation with a friendly, well-educated German classical musician next to me. We soon started to discuss politics and religion. Over the course of the conversation, in response to his questions, I explained I had twice voted against George W. Bush, that I opposed the war in Iraq for many reasons, that I did thought there was an ethical imperative to work to defeat climate change, that I viewed health care as an important ethical and religious issue, that I thought evolution was well-established, and that I am a Christian.

Finally, without any hint of insult intended, and rather a lot of surprise written all over his face, he said:

“Wow. You’re an American, and a Christian, and you’re so…. normal!”

This, it seems to me, has a lot to do with Trump.

Ouch

It felt like a punch to the gut. The day after the election, having known that a man that appeared to stand for everything that honorable people are against won the election, like people all around the world, I was trying to make sense of “how could this happen?” As I’ve watched since, as he stacks government with wealthy cronies with records nearly as colorful as his own, it is easy to feel even more depressed.

Based on how Trump spoke and acted, it would be easy to conclude that the “deplorables” won the day – that he was elected by a contingent of sexists or racists ascendent in power.

But that would be too simple an explanation. This is, after all, the same country that elected Barack Obama twice. There are a many people that voted twice for a black man, and then for Trump. Why? Racism, while doubtless a factor, can’t explain it all.

[Many Americans are] hurt, and they’re scared, and they feel like a lot of the United States just slammed the door in their faces.”

The status quo is not working for people.

Technocratic government by political elites is not working for people. Business as usual is not working for people. Minor tweaks to increasingly arcane systems is not working for people. People are feeling lost in bureaucracy, disaffected by elections that do not present a clear alternate vision, and depressed by a slow slide into increasingly dismal circumstances.

Government is not doing what we want it to do for us. And people are getting left behind. The left in the United States (of which I’m part) has for many years been very concerned about the way blacks and other racial minorities are systematically pushed to the margins of our economy, and how women are pushed out of leadership roles. Those problems are real. But the loss of jobs in the industrial heartland, the inability of a white, rural, working-class man to support his family the way his father supported him, the collapse of once-vibrant communities into poverty and despair: those problems are real too.

The status quo is not working for anyone except for a few lucky, highly-educated people on the coasts. People, honestly, like me, and like many of the other (primarily white and male) people who work in tech. We are one of the few beneficiaries of a system that is failing the vast majority of people in this country.

Russ is, of course, right. The Democrats have been either complicit in policies damaging to many, or ineffective in preventing them. They have often appeared unconcerned with the plight of people outside cities (even if that wasn’t really the case). And it goes deeper.

When’s the last time you visited Kansas?

I live in Kansas. The nearest paved road is about a 3-mile drive from my home. The nearest town, population 600, is a 6-mile drive. My governor — whom I did not vote for — cut taxes on the wealthy so much that our excellent local schools have been struggling for years. But my community is amazing, full of loving and caring people, the sort of people who you know you’ll be living with for 40 years, and so you make sure you get along well with.

I have visited tourist sites in Berlin, enjoyed an opera and a Broadway show in New York, taken a train across the country to Portland, explored San Francisco. I’ve enjoyed all of them. Many rural people do get out and experience the world.

I have been in so many conversations where I try to explain where I live to people that simply cannot fathom it. I have explained how the 18 acres I own is a very small amount where I am. How, yes, I do actually have electricity and Internet. How a bad traffic day is one where I have to wait for three cars to go past before turning onto the paved road. How I occasionally find a bull in my front yard, how I can walk a quarter mile and be at the creek on the edge of my property, how I can get to an airport faster than most New Yorkers and my kids can walk out the front door and play in a spot more peaceful than Central Park, and how all this is way cheaper than a studio apartment in a bad part of San Francisco.

It is rare indeed to see visitors actually traveling to Kansas as a destination. People have no concept of the fact that my mechanic would drop everything and help me get my broken-down car to the shop for no charge, that any number of neighbors or uncles would bring a tractor and come plow the snow off my 1/4-mile driveway out of sheer kindness, that people around here really care for each other in a way you don’t see in a city.

There are people that I know see politics way differently than me, but I know them to be good people. They would also do anything for a person in need, no matter who they are. I may find the people that they vote for to be repugnant, but I cannot say “I’ve looked this person in the eyes and they are nothing but deplorable.”

And so, people in rural areas feel misunderstood. And they are right.

Some perspectives on Trump

As I’ve said, I do find Trump to be deplorable, but not everyone that voted for him is. How, then, do people wind up voting for him?

The New Yorker had an excellent story about a man named Mark Frisbie, owner of a welding and fab shop. The recession had been hard on his business. His wife’s day-care center also closed. Health care was hard to find, and the long, slow decline had spanned politicians of every stripe. Mark and his wife supposedly did everything they were supposed to: they worked hard, were honest, were entrepreneurial, and yet — he had lost his business, his family house, his health coverage, everything. He doesn’t want a handout. He wants to be able to earn a living. Asked who he’d vote for, he said, “Is ‘none of the above’ an option?”

The Washington Post had another insightful article, about a professor from Madison, WI interviewing people in rural areas. She said people would often say: “All the decisions are made in Madison and Milwaukee and nobody’s listening to us. Nobody’s paying attention, nobody’s coming out here and asking us what we think. Decisions are made in the cities, and we have to abide by them.” She pushed back, hard, on the idea that Trump supporters are ignorant, and added that liberals that push that line of thinking are only making the problem worse.

I would agree; seeing all the talk about universities dis-inviting speakers that don’t hew to certain political views doesn’t help either.

And then we have a more recent CNN article: Where Tump support and Obamacare use soar together, explaining in great detail how it can be logical for someone to be on Obamacare but not like it. We can all argue that the Republicans may have as much to do with that as anything, but the problem exists.

“His supporters realize he’s a joke. They do not care. They know he’s authoritarian, nationalist, almost un-American, and they love him anyway, because he disrupts a broken political process and beats establishment candidates who’ve long ignored their interests.

When you’re earning $32,000 a year and haven’t had a decent vacation in over a decade, it doesn’t matter who Trump appoints to the U.N., or if he poisons America’s standing in the world, you just want to win again, whoever the victim, whatever the price.

…

According to the Republican Party, the biggest threat to rural America was Islamic terrorism. According to the Democratic Party it was gun violence. In reality it was prescription drug abuse and neither party noticed until it was too late.”

Are we leaving people out?

All this reminded me of reading about Donald Knuth, the famous computer scientist and something of the father of modern computing, writing about his feelings of trepidation about sharing with his university colleagues that he was working on a project related to the Bible. I am concerned about the complaints about “the PC culture”, because I think it is good that people aren’t making racist or anti-semitic jokes in public anymore. But, as some of these articles point out, in many circles, making fun of Christians and conservatives is still one of the accepted targets. Does that really help anything? (And as a Christian that is liberal, have all of you that aren’t Christians so quickly forgotten how churches like the Episcopals blazed the way for marriage equality many years ago already?)

But they don’t get a free pass

I have found a few things, however, absolutely scary. One was an article from December showing that Trump voters actually changed their views on Russia after Trump became the nominee. Another one from just today was a study on how people reacted when showed inauguration crowd photos.

So where does this leave us? Heading into uncertain times, for sure, but perhaps — just maybe — with a greater understanding of our neighbors.

Perhaps we will all be able to see past the rhetoric and polarization, and understand that there is something, well, normal about each other. Doing that is going to be the only way we can really take our country back.

]]>http://changelog.complete.org/archives/9762-what-is-happening-to-america/feed189762Singing with Kidshttp://changelog.complete.org/archives/9756-singing-with-kids
http://changelog.complete.org/archives/9756-singing-with-kids#respondThu, 22 Dec 2016 01:35:43 +0000http://changelog.complete.org/?p=9756Continue reading Singing with Kids→]]>For four years now, we’ve had a tradition: I go up to the attic one night, make a lot of noise, and pretend to be Santa. The boys don’t think Santa is real, but they get a huge kick out of this anyway.

The other day, this wound up with me singing a duet with my 7-year-old Oliver, and seeing a hugely delighted 10-year-old Jacob.

All last week, the boys had been lobbying for me to “be Santa”. They aren’t going to be able to be here on Christmas day this year, so I thought – why not let them have some fun. I chose one present to give them early too.

So, Saturday night, I said they could get ready for Santa. They found some cookies somewhere, got out some milk. And Oliver wrote this wonderful note to “Santa”:

That is a note I’m going to keep for a long time. He helpfully drew arrows pointing to the milk, cookies, and even the pen. He even started Santa’s reply at the bottom!

So, Saturday night, I snuck up to the attic, pretended to be Santa, and ate some cookies, drank some milk, and wrote Oliver a note. And I left a present.

Jacob has been really getting into music lately, and Laura suggested I find something for the boys. I went looking for something that could record also, and came up with what has got to be a kid’s dream: a karaoke machine.

The particular one I found came with two microphones, a CD player, audio recording onto SD card (though it’s a little dodgy), and a screen for showing words on any music that’s karaoke-enhanced.

Cue gasps of awe and excitement from the boys when we came down in our PJs and sweats at 6:45 Sunday morning to check it out.

Jacob excitedly began exploring all the knobs and options on it (they were particularly fond of the echo feature), while Oliver wanted to sing. So we found one of his favorite Christmas songs, and here he is singing it with me.

When you have a system with a line in, line out, and several microphone jacks, you can get creative. With a few bits of adapters from my attic, the headset I use for amateur radio worked with it perfectly. Add on a little mic extension cord, and pretty soon Oliver was pretending to be an announcer for a football game!

Then, Oliver decided he would act out a football game while Jacob and I were the announcers.

Something tells me there will be much fun had with this over the next while!

Just wait until I show them how to hook up a handheld radio to it in order to make a remotely-activated loudspeaker…

]]>http://changelog.complete.org/archives/9756-singing-with-kids/feed09756Giant Concrete Arrows, Old Maps, and Fascinated Kidshttp://changelog.complete.org/archives/9741-giant-concrete-arrows-old-maps-and-fascinated-kids
http://changelog.complete.org/archives/9741-giant-concrete-arrows-old-maps-and-fascinated-kids#commentsFri, 09 Dec 2016 03:04:28 +0000http://changelog.complete.org/?p=9741Continue reading Giant Concrete Arrows, Old Maps, and Fascinated Kids→]]>Let me set a scene for you. Two children, ages 7 and 10, are jostling for position. There’s a little pushing and shoving to get the best view.

This is pretty typical for siblings this age. But what, you may wonder, are they trying to see? A TV? Video game?

No. Jacob and Oliver were in a library, trying to see a 98-year-old map of the property owners in Township 23, range 1 East, Harvey County, Kansas. And they were super excited about it, somewhat to the astonishment of the research librarian, who I am sure is more used to children jostling for position over the DVDs in the youth section than poring over maps in the non-circulating historical archives!

Every so often, people stumble upon these abandoned arrows and there is a big discussion online. Even Snopes has had to verify their authenticity (verdict: true). Entire websites exist to tracking and locating the remnants of these arrows. And as one of the early air mail routes went through Kansas, every so often people find these arrows around here.

I got the idea that it would be fun to replicate a journey along the old routes. Maybe I’d spot a few old arrows and such. So I started collecting old maps: a Contract Airmail Route #34 (CAM 34) map from 1927, aviation sectionals from 1933 and 1946, etc.

I noticed an odd thing on these maps: the Newton, KS airport was on the other side of the city from its present location, sometimes even several miles outside the city. What was going on?

(1927 Airway Map)

(1946 Wichita sectional)

So one foggy morning, I explained my puzzlement to the boys. I highlighted all the mysteries: were these maps correct? Were there really two Newton airports at one time? How many airports were there, and where were they? Why did they move? What was the story behind them?

And I offered them the chance to be history detectives with me. And oh my goodness, were they ever excited! We had some information from a very helpful person at the Harvey County Historical Museum (thanks Kris!) So we suspected one airport at least was established in 1927. We also had a description of its location, though given in terms of township maps.

So the boys and I made the short drive over to the museum. We reviewed their property maps, though they were all a little older than the time period we needed. We looked through books and at pictures. Oliver pored over a railroad map of Newton from a century ago, fascinated. Jacob was excited to discover on one map that there used to be a train track down the middle of Main Street! I was interested that the present Newton Airport was once known as Wirt Field, rather to my surprise. I somehow suspect most 2nd and 4th graders spend a lot less excited time on their research floor!

Then on to the Newton Public Library to see if they’d have anything more — and that’s when the map that produced all the excitement came out.

It, by itself, didn’t answer the question, but by piecing together a number of pieces of information — newspaper stories, information from the museum, and the maps — we were able to come up with a pretty good explanation, much to their excitement.

Apparently, a man named Tangeman owned a golf course (the “golf links” according to the paper), and around 1927 the city of Newton purchased it, because of all the planes that were landing there. They turned it into a real airport. Later, they bought land east of the city and moved the airport there. However, during World War II, the Navy took over that location, so they built a third airport a few miles west of the city — but moved back to the current east location after the Navy returned that field to them.

Of course, a project like this just opens up all sorts of extra questions: why isn’t it called Wirt Field anymore? What’s the story of Frank Wirt? What led the Navy to take over Newton’s airport? Why did planes start landing on the golf course? Where precisely was the west airport located? How long was it there? (I found an aerial photo from 1956 that looks like it may have a plane in that general area, but it seems later than I’d have expected)

So now I have the boys interested in going to the courthouse with me to research the property records out there. Jacob is continually astounded that we are discovering things that aren’t in Wikipedia, and also excited that he could be the one to add them. To be continued, apparently!

This is morning. Time to fly. Two boys, happy to open the hangar door and get the plane ready.

It’s been a year since I passed the FAA exam and became a pilot. Memories like these are my favorite reminders why I did. It is such fun to see people’s faces light up with the joy of flying a few thousand feet above ground, of the beauty and freedom and peace of the skies.

I’ve flown 14 different passengers in that time; almost every flight I’ve taken has been with people, which I enjoy. I’ve heard “wow” or “beautiful” so many times, and said it myself even more times.

I’ve landed in two state parks, visited any number of wonderful small towns, seen historic sites and placid lakes, ascended magically over forests and plains. I’ve landed at 31 airports in 10 states, flying over 13,000 miles.

Not once have I encountered anyone other than friendly, kind, and outgoing. And why not? After all, we’re working around magic flying carpet machines, right?

(That’s my brother before a flight with me, by the way)

Some weeks it is easy to be glum. This week has been that way for many, myself included. But then, whether you are in the air or on the ground, if you pay attention, you realize we still live in a beautiful world with many wonderful people.

And, in fact, I got a reminder of that this week. Not long after the election, I got in a plane, pushed in the throttle, and started the takeoff roll down a runway in the midst of an Indiana forest. The skies were the best kind of clear blue, and pretty soon I lifted off and could see for miles. Off in the distance, I could see the last cottony remnants of the morning’s fog, lying still in the valleys, surrounding the little farms and houses as if to give them a loving hug. Wow.

Sometimes the flight is bumpy. Sometimes the weather doesn’t cooperate, and it doesn’t happen at all. Sometimes you can fly across four large states and it feels as smooth as glass the whole way.

Whatever happens, at the end of the day, the magic flying carpet machine gets locked up again. We go home, rest our heads on our soft pillows, and if we so choose, remember the beauty we experienced that day.

Really, this post is not about being a pilot. This post is a reminder to pay attention to all that is beautiful in this world. It surrounds us; the smell of pine trees in the forest, the delight in the faces of children, the gentle breeze in our hair, the kind word from a stranger, the very sunrise.

I hope that more of us will pay attention to the moments of clear skies and wind at our back. Even at those moments when we pull the hangar door shut.

]]>http://changelog.complete.org/archives/9736-morning-in-the-skies/feed39736Two Boys, An Airplane, Plus Hundreds of Old Computershttp://changelog.complete.org/archives/9719-two-boys-an-airplane-plus-hundreds-of-old-computers
http://changelog.complete.org/archives/9719-two-boys-an-airplane-plus-hundreds-of-old-computers#commentsTue, 13 Sep 2016 17:03:42 +0000http://changelog.complete.org/?p=9719Continue reading Two Boys, An Airplane, Plus Hundreds of Old Computers→]]>“Was there anything you didn’t like about our trip?”

Jacob’s answer: “That we had to leave so soon!”

That’s always a good sign.

When I first heard about the Vintage Computer Festival Midwest, I almost immediately got the notion that I wanted to go. Besides the TRS-80 CoCo II up in my attic, I also have fond memories of an old IBM PC with CGA monitor, a 25MHz 486, an Alpha also in my attic, and a lot of other computers along the way. I didn’t really think my boys would be interested.

But I mentioned it to them, and they just lit up. They remembered the Youtube videos I’d shown them of old line printers and punch card readers, and thought it would be great fun. I thought it could be a great educational experience for them too — and it was.

It also turned into a trip that combined being a proud dad with so many of my other interests. Quite a fun time.

(Jacob modeling his new t-shirt)

Captain Jacob

Chicago being not all that close to Kansas, I planned to fly us there. If you’re flying yourself, solid flight planning is always important. I had already planned out my flight using electronic tools, but I always carry paper maps with me in the cockpit for backup. I got them out and the boys and I planned out the flight the old-fashioned way.

Here’s Oliver using a scale ruler (with markings for miles corresponding to the scale of the map) and Jacob doing calculating for us. We measured the entire route and came to within one mile of the computer’s calculation for each segment — those boys are precise!

We figured out how much fuel we’d use, where we’d make fuel stops, etc.

The day of our flight, we made it as far as Davenport, Iowa when a chance of bad weather en route to Chicago convinced me to land there and drive the rest of the way. The boys saw that as part of the exciting adventure!

Jacob is always interested in maps, and had kept wanting to use my map whenever we flew. So I dug an old Android tablet out of the attic, put Avare on it (which has aviation maps), and let him use that. He was always checking it while flying, sometimes saying this over his headset: “DING. Attention all passengers, this is Captain Jacob speaking. We are now 45 miles from St. Joseph. Our altitude is 6514 feet. Our speed is 115 knots. We will be on the ground shortly. Thank you. DING”

Here he is at the Davenport airport, still busy looking at his maps:

Every little airport we stopped at featured adults smiling at the boys. People enjoyed watching a dad and his kids flying somewhere together.

Oliver kept busy too. He loves to help me on my pre-flight inspections. He will report every little thing to me – a scratch, a fleck of paint missing on a wheel cover, etc. He takes it seriously. Both boys love to help get the plane ready or put it away.

The Computers

Jacob quickly gravitated towards a few interesting things. He sat for about half an hour watching this old Commodore plotter do its thing (click for video):

His other favorite thing was the phones. Several people had brought complete analog PBXs with them. They used them to demonstrate various old phone-related hardware; one had several BBSs running with actual modems, another had old answering machines and home-security devices. Jacob learned a lot about phones, including how to operate a rotary-dial phone, which he’d never used before!

Oliver was drawn more to the old computers. He was fascinated by the IBM PC XT, which I explained was just about like a model I used to get to use sometimes. They learned about floppy disks and how computers store information.

He hadn’t used joysticks much, and found Pong (“this is a soccer game!”) interesting. Somebody has also replaced the guts of a TRS-80 with a Raspberry Pi running a SNES emulator. This had thoroughly confused me for a little while, and excited Oliver.

Jacob enjoyed an old TRS-80, which, through a modern Ethernet interface and a little computation help in AWS, provided an interface to Wikipedia. Jacob figured out the text-mode interface quickly. Here he is reading up on trains.

I had no idea that Commodore made a lot of adding machines and calculators before they got into the home computer business. There was a vast table with that older Commodore hardware, too much to get on a single photo. But some of the adding machines had their covers off, so the boys got to see all the little gears and wheels and learn how an adding machine can do its printing.

And then we get to my favorite: the big iron. Here is a VAX — a working VAX. When you have a computer that huge, it’s easier for the kids to understand just what something is.

When we encountered the table from the Glenside Color Computer Club, featuring the good old CoCo IIs like what I used as a kid (and have up in my attic), I pointed out to the boys that “we have a computer just like this that can do these things” — and they responded “wow!” I think they are eager to try out floppy disks and disk BASIC now.

Some of my favorites were the old Unix systems, which are a direct ancestor to what I’ve been working with for decades now. Here’s AT&T System V release 3 running on its original hardware:

And there were a couple of Sun workstations there, making me nostalgic for my college days. If memory serves, this one is actually running on m68k in the pre-Sparc days:

Returning home

After all the excitement of the weekend, both boys zonked out for awhile on the flight back home. Here’s Jacob, sleeping with his maps still up.

As we were nearly home, we hit a pocket of turbulence, the kind that feels as if the plane is dropping a bit (it’s perfectly normal and safe; you’ve probably felt that on commercial flights too). I was a bit concerned about Oliver; he is known to get motion sick in cars (and even planes sometimes). But what did I hear from Oliver?

“Whee! That was fun! It felt like a roller coaster! Do it again, dad!”

]]>http://changelog.complete.org/archives/9719-two-boys-an-airplane-plus-hundreds-of-old-computers/feed39719Easily Improving Linux Security with Two-Factor Authenticationhttp://changelog.complete.org/archives/9707-easily-improving-linux-security-with-two-factor-authentication
http://changelog.complete.org/archives/9707-easily-improving-linux-security-with-two-factor-authentication#commentsTue, 09 Aug 2016 22:23:35 +0000http://changelog.complete.org/?p=9707Continue reading Easily Improving Linux Security with Two-Factor Authentication→]]>2-Factor Authentication (2FA) is a simple way to help improve the security of your systems. It restricts the scope of damage if a machine is compromised. If, for instance, you have a security token or authenticator app on your phone that is required for ssh to a remote machine, then even if every laptop you use to connect to the remote is totally owned, an attacker cannot establish a new ssh session on their own.

There are a lot of tutorials out there on the Internet that get you about halfway there, so here is some more detail.

Background

In this article, I will be focusing on authentication in the style of Google Authenticator, which is a special case of OATH HOTP or TOTP. You can use the Google Authenticator app, FreeOTP, or a hardware token like Yubikey to generate tokens with this. They are all 100% compatible with Google Authenticator and libpam-google-authenticator.

The basic idea is that there is a pre-shared secret key. At each login, a different and unique token is required, which is generated based on the pre-shared secret key and some other information. With TOTP, the “other information” is the current time, implying that both machines must be reasably well in-sync time-wise. With HOTP, the “other information” is a count of the number of times the pre-shared key has been used. Both typically have a “window” on the server side that can let times within a certain number of seconds, or a certain number of login accesses, work.

The beauty of this system is that after the initial setup, no Internet access is required on either end to validate the key (though TOTP requires both ends to be reasonably in sync time-wise).

The basics: user account setup and ssh authentication

You can start with the basics by reading one of these articles: one, two, three. Debian/Ubuntu users will find both the pam module and the user account setup binary in libpam-google-authenticator.

For many, you can stop there. You’re done. But if you want to kick it up a notch, read on:

Enhancement 1: Requiring 2FA even when ssh public key auth is used

Let’s consider a scenario in which your system is completely compromised. Unless your ssh keys are also stored in something like a Yubikey Neo, they could wind up being compromised as well – if someone can read your files and sniff your keyboard, your ssh private keys are at risk.

So we can configure ssh and PAM so that a OTP token is required even for this scenario.

First off, in /etc/ssh/sshd_config, we want to change or add these lines:

This forces all authentication to pass two verification methods in ssh: publickey and keyboard-interactive. All users will have to supply a public key and then also pass keyboard-interactive auth. Normally keyboard-interactive auth prompts for a password, but we can change /etc/pam.d/sshd on this. I added this line at the very top of /etc/pam.d/sshd:

This basically makes Google Authenticator both necessary and sufficient for keyboard-interactive in ssh. That is, whenever the system wants to use keyboard-interactive, rather than prompt for a password, it instead prompts for a token. Note that any user that has not set up google-authenticator already will be completely unable to ssh into their account.

Enhancement 1, variant 2: Allowing automated processes to root

On many of my systems, I have ~root/.ssh/authorized_keys set up to permit certain systems to run locked-down commands for things like backups. These are automated commands, and the above configuration will break them because I’m not going to be typing in codes at 3AM.

If you are very restrictive about what you put in root’s authorized_keys, you can exempt the root user from the 2FA requirement in ssh by adding this to sshd_config:

Match User root
AuthenticationMethods publickey

This says that the only way to access the root account via ssh is to use the authorized_keys file, and no 2FA will be required in this scenario.

Enhancement 1, variant 2: Allowing non-pubkey auth

On some multiuser systems, some users may still want to use password auth rather than publickey auth. There are a few ways we can support that:

Users without public keys will have to supply a OTP and a password, while users with public keys will have to supply public key, OTP, and a password

Users without public keys will have to supply OTP or a password, while users with public keys will have to supply public key, OTP, or a password

Users without public keys will have to supply OTP and a password, while users with public keys only need to supply the public key

The third option is covered in any number of third-party tutorials. To enable options 1 or 2, you’ll need to put this in sshd_config:

This means that to authenticate, you need to pass either publickey and then keyboard-interactive auth, or just keyboard-interactive auth.

Then in /etc/pam.d/sshd, you put this:

auth required pam_google_authenticator.so

As a sub-variant for option 1, you can add nullok to here to permit auth from people that do not have a Google Authenticator configuration.

Or for option 2, change “required” to “sufficient”. You should not add nullok in combination with sufficient, because that could let people without a Google Authenticator config authenticate completely without a password at all.

Enhancement 2: Configuring su

A lot of other tutorials stop with ssh (and maybe gdm) but forget about the other ways we authenticate or change users on a system. su and sudo are the two most important ones. If your root password is compromised, you don’t want anybody to be able to su to that account without having to supply a token. So you can set up google-authenticator for root.

Then, edit /etc/pam.d/su and insert this line after the pam_rootok.so line:

auth required pam_google_authenticator.so nullok

The reason you put this after pam_rootok.so is because you want to be able to su from root to any account without having to input a token. We add nullok to the end of this, because you may want to su to accounts that don’t have tokens. Just make sure to configure tokens for the root account first.

Enhancement 3: Configuring sudo

This one is similar to su, but a little different. This lets you, say, secure the root password for sudo.

Normally, you might sudo from your user account to root (if so configured). You might have sudo configured to require you to enter in your own password (rather than root’s), or to just permit you to do whatever you want as root without a password.

Our first step, as always, is to configure PAM. What we do here depends on your desired behavior: do you want to require someone to supply both a password and a token, or just a token, or require a token? If you want to require a token, put this at the top of /etc/pam.d/sudo:

If you want to require a token and a password, change the bracketed string to “required”, and if you want a token or a password, change it to “sufficient”. As before, if you want to permit people without a configured token to proceed, add “nullok”, but do not use that with “sufficient” or the bracketed example here.

Now here comes the fun part. By default, if a user is required to supply a password to sudo, they are required to supply their own password. That does not help us here, because a user logged in to the system can read the ~/.google_authenticator file and easily then supply tokens for themselves. What you want to do is require them to supply root’s password. Here’s how I set that up in sudoers:

Defaults:jgoerzen rootpw
jgoerzen ALL=(ALL) ALL

So now, with the combination of this and the PAM configuration above, I can sudo to the root user without knowing its password — but only if I can supply root’s token. Pretty slick, eh?

First, the libpam-google-authenticator module supports putting the Google Authenticator files in different locations and having them owned by a certain user. You could use this to, for instance, lock down all secret keys to be readable only by the root user. This would prevent users from adding, changing, or removing their own auth tokens, but would also let you do things such as reusing your personal token for the root account without a problem.

Also, the pam-oath module does much of the same things as the libpam-google-authenticator module, but without some of the help for setup. It uses a single monolithic root-owned password file for all accounts.

There is an oathtool that can be used to generate authentication codes from the command line.

And so began a “trip” aboard our hotel train in Indianapolis, conducted by our very own Jacob and Oliver.

Because, well, what could be more fun than spending a few days in the world’s only real Pullman sleeping car, on its original service track, inside a hotel?

We were on a family vacation to Indianapolis, staying in what two railfan boys were sure to enjoy: a hotel actually built into part of the historic Indianapolis Union Station complex. This is the original train track and trainshed. They moved in the Pullman cars, then built the hotel around them. Jacob and Oliver played for hours, acting as conductors and engineers, sending their “train” all across the country to pick up and drop off passengers.

Opa!

Have you ever seen a kid’s face when you introduce them to something totally new, and they think it is really exciting, but a little scary too?

That was Jacob and Oliver when I introduced them to saganaki (flaming cheese) at a Greek restaurant. The conversation went a little like this:

“Our waitress will bring out some cheese. And she will set it ON FIRE — right by our table!”

“Will it burn the ceiling?”

“No, she’ll be careful.”

“Will it be a HUGE fire?”

“About a medium-sized fire.”

“Then what will happen?”

“She’ll yell ‘OPA!’ and we’ll eat the cheese after the fire goes out.”

“Does it taste good?”

“Oh yes. My favorite!”

It turned out several tables had ordered saganaki that evening, so whenever I saw it coming out, I’d direct their attention to it. Jacob decided that everyone should call it “opa” instead of saganaki because that’s what the waitstaff always said. Pretty soon whenever they’d see something appear in the window from the kitchen, there’d be craning necks and excited jabbering of “maybe that’s our opa!”

And when it finally WAS our “opa”, there were laughs of delight and I suspect they thought that was the best cheese ever.

Giggling Elevators

Fun times were had pressing noses against the glass around the elevator. Laura and I sat on a nearby sofa while Jacob and Oliver sat by the elevators, anxiously waiting for someone to need to go up and down. They point and wave at elevators coming down, and when elevator passengers waved back, Oliver would burst out giggling and run over to Laura and me with excitement.

Some history

We got to see the grand hall of Indianapolis Union Station — what a treat to be able to set foot in this magnificent, historic space, the world’s oldest union station. We even got to see the office where Thomas Edison worked, and as a hotel employee explained, was fired for doing too many experiments on the job.

Water and walkways

Indy has a system of elevated walkways spanning quite a section of downtown. It can be rather complex navigating them, and after our first day there, I offered to let Jacob and Oliver be the leaders. Boy did they take pride in that! They stopped to carefully study maps and signs, and proudly announced “this way” or “turn here” – and were usually correct.

And it was the same in the paddleboat we took down the canal. Both boys wanted to be in charge of steering, and we only scared a few other paddleboaters.

Fireworks

Our visit ended with the grand fireworks show downtown, set off from atop a skyscraper. I had been scouting for places to watch from, and figured that a bridge-walkway would be great. A couple other families had that thought too, and we all watched the 20-minute show in the drizzle.

Loving brothers

By far my favorite photo from the week is this one, of Jacob and Oliver asleep, snuggled up next to each other under the covers. They sure are loving and caring brothers, and had a great time playing together.

]]>http://changelog.complete.org/archives/9703-all-aboard/feed119703Building a home firewall: review of pfsensehttp://changelog.complete.org/archives/9700-building-a-home-firewall-review-of-pfsense
http://changelog.complete.org/archives/9700-building-a-home-firewall-review-of-pfsense#commentsMon, 18 Jul 2016 21:34:02 +0000http://changelog.complete.org/?p=9700Continue reading Building a home firewall: review of pfsense→]]>For some time now, I’ve been running OpenWRT on an RT-N66U device. I initially set that because I had previously been using my Debian-based file/VM server as a firewall, and this had some downsides: every time I wanted to reboot that, Internet for the whole house was down; shorewall took a fair bit of care and feeding; etc.

I’ve been having indications that all is not well with OpenWRT or the N66U in the last few days, and some long-term annoyances prompted me to search out a different solution. I figured I could buy an embedded x86 device, slap Debian on it, and be set.

As expected, with hardware like that to work with, it was a lot more capable than OpenWRT and had more features. However, I encountered a number of surprising issues.

The biggest annoyance was that the system wouldn’t allow me to set up a static DHCP entry with the same IP for multiple MAC addresses. This is a very simple configuration in the underlying DHCP server, and OpenWRT permitted it without issue. It is quite useful so my laptop has the same IP whether connected by wifi or Ethernet, and I have used it for years with no issue. Googling it a bit turned up some rather arrogant pfsense people saying that this is “broken” and poor design, and that your wired and wireless networks should be on different VLANs anyhow. They also said “just give it the same hostname for the different IPs” — but it rejects this too. Sigh. I discovered, however, that downloading the pfsense backup XML file, editing the IP within, and re-uploading it gets me what I want with no ill effects!

So then I went to set up DNS. I tried to enable the “DNS Forwarder”, but it wouldn’t let me do that while the “DNS Resolver” was still active. Digging in just a bit, it appears that the DNS Forwarder and DNS Resolver both provide forwarding and resolution features; they just have different underlying implementations. This is not clear at all in the interface.

Next stop: traffic shaping. Since I use VOIP for work, this is vitally important for me. I dove in, and found a list of XML filenames for wizards: one for “Dedicated Links” and another for “Multiple Lan/Wan”. Hmmm. Some Googling again turned up that everyone suggests using the “Multiple Lan/Wan” wizard. Fine. I set it up, and notice that when I start an upload, my download performance absolutely tanks. Some investigation shows that outbound ACKs aren’t being handled properly. The wizard had created a qACK queue, but neglected to create a packet match rule for it, so ACKs were not being dealt with appropriately. Fixed that with a rule of my own design, and now downloads are working better again. I also needed to boost the bandwidth allocated to qACK (setting it to 25% seemed to do the trick).

Then there was the firewall rules. The “interface” section is first-match-wins, whereas the “floating” section is last-match-wins. This is rather non-obvious.

Getting past all the interface glitches, however, the system looks powerful, solid, and well-engineered under the hood, and fairly easy to manage.

]]>http://changelog.complete.org/archives/9700-building-a-home-firewall-review-of-pfsense/feed99700A great day for a flight with the boyshttp://changelog.complete.org/archives/9695-a-great-day-for-a-flight-with-the-boys
http://changelog.complete.org/archives/9695-a-great-day-for-a-flight-with-the-boys#respondTue, 28 Jun 2016 03:57:39 +0000http://changelog.complete.org/?p=9695Continue reading A great day for a flight with the boys→]]>I tend to save up my vacation time to use in summer for family activities, and today was one of those days.

Yesterday, Jacob and Oliver enjoyed planning what they were going to do with me. They ruled out all sorts of things nearby, but they decided they would like to fly to Ponca City, explore the oil museum there, then eat at Enrique’s before flying home.

Of course, it is not particularly hard to convince me to fly somewhere. So off we went today for some great father-son time.

The weather on the way was just gorgeous. We cruised along at about a mile above ground, which gave us pleasantly cool air through the vents and a smooth ride. Out in the distance, a few clouds were trying to form.

Whether I’m flying or driving, a pilot is always happy to pass a small airport. Here was the Winfield, KS airport (KWLD):

This is a beautiful time of year in Kansas. The freshly-cut wheat fields are still a vibrant yellow. Other crops make a bright green, and colors just pop from the sky. A camera can’t do it justice.

They enjoyed the museum, and then Oliver wanted to find something else to do before we returned to the airport for dinner. A little exploring yielded the beautiful and shady Garfield Park, complete with numerous old stone bridges.

Of course, the hit of any visit to Enrique’s is their “ice cream tacos” (sopapillas with ice cream). Here is Oliver polishing off his.

They had both requested sightseeing from the sky on our way back, but both fell asleep so we opted to pass on that this time. Oliver slept through the landing, and I had to wake him up when it was time to go. I always take it as a compliment when a 6-year-old sleeps through a landing!

Most small airports have a bowl of candy setting out somewhere. Jacob and Oliver have become adept at finding them, and I will usually let them “talk me into” a piece of candy at one of them. Today, after we got back, they were intent at exploring the small gift shop back home, and each bought a little toy helicopter for $1.25. They may have been too tired to enjoy it though.

They’ve been in bed for awhile now, and I’m still smiling about the day. Time goes fast when you’re having fun, and all three of us were. It is fun to see them inheriting my sense of excitement at adventure, and enjoying the world around them as they go.

The lady at the museum asked how we had heard about them, and noticed I drove up in an airport car (most small airports have an old car you can borrow for a couple hours for free if you’re a pilot). I told the story briefly, and she said, “So you flew out to this small town just to spend some time here?” “Yep.” “Wow, that’s really neat. I don’t think we’ve ever had a visitor like you before.” Then she turned to the boys and said, “You boys are some of the luckiest kids in the world.”

And I can’t help but feel like the luckiest dad in the world.

]]>http://changelog.complete.org/archives/9695-a-great-day-for-a-flight-with-the-boys/feed09695I’m switching from git-annex to Syncthinghttp://changelog.complete.org/archives/9692-im-switching-from-git-annex-to-syncthing
http://changelog.complete.org/archives/9692-im-switching-from-git-annex-to-syncthing#commentsMon, 27 Jun 2016 13:02:36 +0000http://changelog.complete.org/?p=9692Continue reading I’m switching from git-annex to Syncthing→]]>I wrote recently about using git-annex for encrypted sync, but due to a number of issues with it, I’ve opted to switch to Syncthing.

Anyhow, git-annex is still an awesome tool with a lot of use cases, but I’m concluding that live sync to an encrypted git remote isn’t quite there yet enough for me.

So I looked for alternatives. My main criteria were supporting live sync (via inotify or similar) and not requiring the files to be stored unencrypted on a remote system (my local systems all use LUKS). I found Syncthing met these requirements.

Syncthing is pretty interesting in that, like git-annex, it doesn’t require a centralized server at all. Rather, it forms basically a mesh between your devices. Its concept is somewhat similar to the proprietary Bittorrent Sync — basically, all the nodes communicate about what files and chunks of files they have, and the changes that are made, and immediately propagate as much as possible. Unlike, say, Dropbox or Owncloud, Syncthing can actually support simultaneous downloads from multiple remotes for optimum performance when there are many changes.

Syncthing is particularly adept at figuring out ways for the nodes to communicate with each other. It begins by broadcasting on the local network, so known nearby nodes can be found directly. The Syncthing folks also run a discovery server (though you can use your own if you prefer) that lets nodes find each other on the Internet. Syncthing will attempt to use UPnP to configure firewalls to let it out, but if that fails, the last resort is a traffic relay server — again, a number of volunteers host these online, but you can run your own if you prefer.

Each node in Syncthing has an RSA keypair, and what amounts to part of the public key is used as a globally unique node ID. The initial link between nodes is accomplished by pasting the globally unique ID from one node into the “add node” screen on the other; the user of the first node then must accept the request, and from that point on, syncing can proceed. The data is all transmitted encrypted, of course, so interception will not cause data to be revealed.

There is an ITP open for Syncthing in Debian, but until then, their apt repo works fine. For syncthing-gtk, the trusty version of the webupd8 PPD works in Jessie (though be sure to pin it to a low priority if you don’t want it replacing some unrelated Debian packages).

]]>http://changelog.complete.org/archives/9692-im-switching-from-git-annex-to-syncthing/feed89692Mud, Airplanes, Arduino, and Funhttp://changelog.complete.org/archives/9682-mud-airplanes-arduino-and-fun
http://changelog.complete.org/archives/9682-mud-airplanes-arduino-and-fun#commentsThu, 16 Jun 2016 04:00:46 +0000http://changelog.complete.org/?p=9682Continue reading Mud, Airplanes, Arduino, and Fun→]]>The last few weeks have been pretty hectic in their way, but I’ve also had the chance to take some time off work to spend with family, which has been nice.

Memorial Day: breakfast and mud

For Memorial Day, I decided it would be nice to have a cookout for breakfast rather than for dinner. So we all went out to the fire ring. Jacob and Oliver helped gather kindling for the fire, while Laura chopped up some vegetables. Once we got a good fire going, I cooked some scrambled eggs in a cast iron skillet, mixed with meat and veggies. Mmm, that was tasty.

Then we all just lingered outside. Jacob and Oliver enjoyed playing with the cats, and the swingset, and then…. water. They put the hose over the slide and made a “water slide” (more mud slide maybe).

Then we got out the water balloon fillers they had gotten recently, and they loved filling up water balloons. All in all, we all just enjoyed the outdoors for hours.

Flying to Petit Jean, Arkansas

Somehow, neither Laura nor I have ever really been to Arkansas. We figured it was about time. I had heard wonderful things about Petit Jean State Park from other pilots: it’s rather unique in that it has a small airport right in the park, a feature left over from when Winthrop Rockefeller owned much of the mountain.

And what a beautiful place it was! Dense forests with wonderful hiking trails, dotted with small streams, bubbling springs, and waterfalls all over; a nice lake, and a beautiful lodge to boot. Here was our view down into the valley at breakfast in the lodge one morning:

And here’s a view of one of the trails:

The sunset views were pretty nice, too:

And finally, the plane we flew out in, parked all by itself on the ramp:

It was truly a relaxing, peaceful, re-invigorating place.

Flying to Atchison

Last weekend, Laura and I decided to fly to Atchison, KS. Atchison is one of the oldest cities in Kansas, and has quite a bit of history to show off. It was fun landing at the Amelia Earhart Memorial Airport in a little Cessna, and then going to three museums and finding lunch too.

Of course, there is the Amelia Earhart Birthplace Museum, which is a beautifully-maintained old house along the banks of the Missouri River.

I was amused to find this hanging in the county historical society museum:

One fascinating find is a Regina Music Box, popular in the late 1800s and early 1900s. It operates under the same principles as those that you might see that are cylindrical. But I am particular impressed with the effort that would go into developing these discs in the pre-computer era, as of course the holes at the outer edge of the disc move faster than the inner ones. It would certainly take a lot of careful calculation to produce one of these. I found this one in the Cray House Museum:

An Arduino Project with Jacob

One day, Jacob and I got going with an Arduino project. He wanted flashing blue lights for his “police station”, so we disassembled our previous Arduino project, put a few things on the breadboard, I wrote some code, and there we go. Then he noticed an LCD in my Arduino kit. I hadn’t ever gotten around to using it yet, and of course he wanted it immediately. So I looked up how to connect it, found an API reference, and dusted off my C skills (that was fun!) to program a scrolling message on it. Here is Jacob showing it off:

]]>http://changelog.complete.org/archives/9682-mud-airplanes-arduino-and-fun/feed39682How git-annex replaces Dropbox + encfs with untrusted providershttp://changelog.complete.org/archives/9669-how-git-annex-replaces-dropbox-encfs-with-untrusted-providers
http://changelog.complete.org/archives/9669-how-git-annex-replaces-dropbox-encfs-with-untrusted-providers#commentsMon, 06 Jun 2016 14:38:56 +0000http://changelog.complete.org/?p=9669Continue reading How git-annex replaces Dropbox + encfs with untrusted providers→]]>git-annex has been around for a long time, but I just recently stumbled across some of the work Joey has been doing to it. This post isn’t about it’s traditional roots in git or all the features it has for partial copies of large data sets, but rather for its live syncing capabilities like Dropbox. It takes a bit to wrap your head around, because git-annex is just a little different from everything else. It’s sort of like a different-colored smell.

The git-annex wiki has a lot of great information — both low-level reference and a high-level 10-minute screencast showing how easy it is to set up. I found I had to sort of piece together the architecture between those levels, so I’m writing this all down hoping it will benefit others that are curious.

Ir you just want to use it, you don’t need to know all this. But I like to understand how my tools work.

Overview

git-annex lets you set up a live syncing solution that requires no central provider at all, or can be used with a completely untrusted central provider. Depending on your usage pattern, this central provider could require only a few MBs of space even for repositories containing gigabytes or terabytes of data that is kept in sync.

Let’s take a look at the high-level architecture of the tool. Then I’ll illustrate how it works with some scenarios.

Three Layers

Fundamentally, git-annex takes layers that are all combined in Dropbox and separates them out. There is the storage layer, which stores the literal data bytes that you are interested in. git-annex indexes the data in storage by a hash. There is metadata, which is for things like a filename-to-hash mapping and revision history. And then there is an optional layer, which is live signaling used to drive the real-time syncing.

git-annex has several modes of operation, and the one that enables live syncing is called the git-annex assistant. It runs as a daemon, and is available for Linux/POSIX platforms, Windows, Mac, and Android. I’ll be covering it here.

The storage layer

The storage layer simply is blobs of data. These blobs are indexed by a hash, and can be optionally encrypted at rest at remote backends. git-annex has a large number of storage backends; some examples include rsync, a remote machine with git-annex on it that has ssh installed, WebDAV, S3, Amazon Glacier, removable USB drive, etc. There’s a huge list.

One of the git-annex features is that each client knows the state of each storage repository, as well as the capability set of each storage repository. So let’s say you have a workstation at home and a laptop you take with you to work or the coffee shop. You’d like changes on one to be instantly recognized on another. With something like Dropbox or OwnCloud, every file in the set you want synchronized has to reside on a server in the cloud. With git-annex, it can be configured such that the server in the cloud only contains a copy of a file until every client has synced it up, at which point it gets removed. Think about it – that is often what you want anyhow, so why maintain an unnecessary copy after it’s synced everywhere? (This behavior is, of course, configurable.) git-annex can also avoid storing in the cloud entirely if the machines are able to reach each other directly at least some of the time.

The metadata layer

Metadata about your files includes a mapping from the file names to the storage location (based on hashes), change history, and information about the status of each machine that participates in the syncing. On your clients, git-annex stores this using git. This detail is very useful to some, and irrelevant to others.

Some of the git-annex storage backends can support only storage (S3, for instance). Some can support both storage and metadata (rsync, ssh, local drives, etc.) You can even configure a backend to support only metadata (more on why that may be useful in a bit). When you are working with a git-backed repository for git-annex, it can hold data, metadata, or both.

So, to have a working sync system, you must have a way to transport both the data and the metadata. The transport for the metadata is generally rsync or git, but it can also be XMPP in which Git changesets are basically wrapped up in XMPP presence messages. Joey says, however, that there are some known issues with XMPP servers sometimes dropping or reordering some XMPP messages, so he doesn’t encourage that method currently.

The live signaling layer

So once you have your data and metadata, you can already do syncs via git annex sync --contents. But the real killer feature here will be automatic detection of changes, both on the local and the remote. To do that, you need some way of live signaling. git-annex supports two methods.

The first requires ssh access to a remote machine where git-annex is installed. In this mode of operation, when the git-annex assistant fires up, it opens up a persistent ssh connection to the remote and runs the git-annex-shell over there, which notifies it of changes to the git metadata repository. When a change is detected, a sync is initiated. This is considered ideal.

A substitute can be XMPP, and git-annex actually converts git commits into a form that can be sent over XMPP. As I mentioned above, there are some known reliability issues with this and it is not the recommended option.

Encryption

When it comes to encryption, you generally are concerned about all three layers. In an ideal scenario, the encryption and decryption happens entirely on the client side, so no service provider ever has any details about your data.

The live signaling layer is encrypted pretty trivially; the ssh sessions are, of course, encrypted and TLS support in XMPP is pervasive these days. However, this is not end-to-end encryption; those messages are decrypted by the service provider, so a service provider could theoretically spy on metadata, which may include change times and filenames, though not the contents of files themselves.

The data layer also can be encrypted very trivially. In the case of the “dumb” backends like S3, git-annex can use symmetric encryption or a gpg keypair and all that ever shows up on the server are arbitrarily-named buckets.

You can also use a gcrypt-based git repository. This can cover both data and metadata — and, if the target also has git-annex installed, the live signalling layer. Using a gcrypt-based git repository for the metadata and live signalling is the only way to accomplish live syncing with 100% client-side encryption.

All of these methods are implemented in terms of gpg, and can support symmetric of public-key encryption.

It should be noted here that the current release versions of git-annex need a one-character patch in order to fix live syncing with a remote using gcrypt. For those of you running jessie, I recommend the version in jessie-backports, which is presently 5.20151208. For your convenience, I have compiled an amd64 binary that can drop in over /usr/bin/git-annex if you have this version. You can download it and a gpg signature for it. Note that you only need this binary on the clients; the server can use the version from jessie-backports without issue.

Putting the pieces together: some scenarios

Now that I’ve explained the layers, let’s look at how they fit together.

Scenario 1: Central server

In this scenario, you might have a workstation and a laptop that sync up with each other by way of a central server that also has a full copy of the data. This is the scenario that most closely resembles Dropbox, box, or OwnCloud.

Here you would basically follow the steps in the git-assistant screencast: install git-annex on a server somewhere, and point your clients to it. If you want full end-to-end encryption, I would recommend letting git-annex generate a gpg keypair for you, which you would then need to copy to both your laptop and workstation (but not the server).

Every change you make locally will be synced to the server, and then from the server to your other PC. All three systems would be configured in the “client” transfer group.

Scenario 1a: Central server without a full copy of the data

In this scenario, everything is configured the same except the central server is configured with the “transfer” transfer group. This means that the actual data synced to it is deleted after it has been propagated to all clients. Since git-annex can verify which repository has received a copy of which data, it can easily enough delete the actual file content from the central server after it has been copied to all the clients. Many people use something like Dropbox or OwnCloud as a multi-PC syncing solution anyhow, so once the files have been synced everywhere, it makes sense to remove them from the central server.

This is often a good ideal for people. There are some obvious downsides that are sometimes relevant. For instance, to add a third sync client, it must be able to initially copy down from one of the existing clients. Or, if you intend to access the data from a device such as a cell phone where you don’t intend for it to have a copy of all data all the time, you won’t have as convenient way to download your data.

Scenario 1b: Split data/metadata central servers

Imagine that you have a shell or rsync account on some remote system where you can run git-annex, but don’t have much storage space. Maybe you have a cheap VPS or shell account somewhere, but it’s just not big enough to hold your data.

The answer to this would be to use this shell or rsync account for the metadata, but put the data elsewhere. You could, for instance, store the data in Amazon S3 or Amazon Glacier. These backends aren’t capable of storing the git-annex metadata, so all you need is a shell or rsync account somewhere to sync up the metadata. (Or, as below, you might even combine a fully distributed approach with this.) Then you can have your encrypted data pushed up to S3 or some such service, which presumably will grow to whatever size you need.

Scenario 2: Fully distributed

Like git itself, git-annex does not actually need a central server at all. If your different clients can reach each other directly at least some of the time, that is good enough. Of course, a given client will not be able to do fully automatic live sync unless it can reach at least one other client, so changes may not propagate as quickly.

You can simply set this up by making ssh connections available between your clients. git-annex assistant can automatically generate appropriate ~/.ssh/authorized_keys entries for you.

Scenario 2a: Fully distributed with multiple disconnected branches

You can even have a graph of connections available. For instance, you might have a couple machines at home and a couple machines at work with no ability to have a direct connection between them (due to, say, firewalls). The two machines at home could sync with each other in real-time, as could the two machines at work. git-annex also supports things like USB drives as a transport mechanism, so you could throw a USB drive in your pocket each morning, pop it in to one client at work, and poof – both clients are synced up over there. Repeat when you get home in the evening, and you’re synced there. The USB drive’s repository can, of course, be of the “transport” type so data is automatically deleted from it once it’s been synced everywhere.

Scenario 3: Hybrid

git-annex can support LAN sync even if you have a central server. If your laptop, say, travels around but is sometimes on the same LAN as your PC, git-annex can easily sync directly between the two when they are reachable, saving a round-trip to the server. You can assign a cost to each remote, and git-annex will always try to sync first to the lowest-cost path that is available.

Drawbacks of git-annex

There are some scenarios where git-annex with the assistant won’t be as useful as one of the more traditional instant-sync systems.

The first and most obvious one is if you want to access the files without the git-annex client. For instance, many of the other tools let you generate a URL that you can email to people, and then they can download files without any special client software. This is not directly possible with git-annex. You could, of course, make something like a public_html directory be managed with git-annex, but it wouldn’t provide things like obfuscated URLs, password-protected sharing, time-limited sharing, etc. that you get with other systems. While you can share your repositories with others that have git-annex, you can’t share individual subdirectories; for a given repository, it is all or nothing.

The Android client for git-annex is a pretty interesting thing: it is mostly a small POSIX environment, providing a terminal, git, gpg, and the same web interface that you get on a standalone machine. This means that the git-annex Android client is fully functional compared to a desktop one. It also has a quick setup process for syncing off your photos/videos. On the other hand, the integration with the Android ecosystem is poor compared to most other tools.

Other git-annex features

git-annex has a lot to offer besides the git-annex assistant. Besides the things I’ve already mentioned, any given git-annex repository — including your client repository — can have a partial copy of the full content. Say, for instance, that you set up a git-annex repository for your music collection, which is quite large. You want some music on your netbook, but don’t have room for it all. You can tell git-annex to get or drop files from the netbook’s repository without deleting them remotely. git-annex has quite a few ways to automate and configure this, including making sure that at least a certain number of copies of a file exist in your git-annex ecosystem.

Conclusion

I initially started looking at git-annex due to the security issues with encfs, and the difficulty with setting up ecryptfs in this way. (I had been layering encfs atop OwnCloud). git-annex certainly ticks the box for me security-wise, and obviously anything encrypted with encfs wasn’t going to be shared with others anyhow. I’ll be using git-annex more in the future, I’m sure.

]]>http://changelog.complete.org/archives/9669-how-git-annex-replaces-dropbox-encfs-with-untrusted-providers/feed19669That was satisfyinghttp://changelog.complete.org/archives/9665-that-was-satisfying
http://changelog.complete.org/archives/9665-that-was-satisfying#respondMon, 30 May 2016 13:29:28 +0000http://changelog.complete.org/?p=9665It’s been awhile due to all sorts of other stuff going on. Nice to see this clogging my inbox again:

It really is satisfying to close bugs!

]]>http://changelog.complete.org/archives/9665-that-was-satisfying/feed09665The Simple Joys of the Plainshttp://changelog.complete.org/archives/9659-the-simple-joys-of-the-plains
http://changelog.complete.org/archives/9659-the-simple-joys-of-the-plains#commentsSun, 08 May 2016 16:24:36 +0000http://changelog.complete.org/?p=9659Continue reading The Simple Joys of the Plains→]]>We love to go exploring as a family. Last year, we gave Jacob and Oliver a theme: “find places older than Grandpa.” They got creative really quick, realizing that any state park counts (“dirt is older than grandpa!”) as did pretty much any museum. Probably our hit from last year was the visit to the tunnels under Ellinwood, KS.

Beatrice, NE

This year, our theme is “places we can fly to”. A couple of weeks ago, Laura had a conference in the beautiful small town of Beatrice, NE. So all four of us flew up, and Jacob, Oliver, and I found fun activities while Laura was at her conference.

We walked around Beatrice a bit, and I noticed this rails-to-trails area. Jacob and Oliver were immediately interested (since it was railroad-related). They quickly turned it into a game of kick-the-dandelion, trying to kick dandelions off their stems and see how high in the air they could get them. The answer: pretty high.

Of course, you can’t go wrong with swimming. Here’s Oliver getting ready for some swimming.

Right near Beatrice is the Homestead National Monument. Of course, the bales decorated like a minion got their attention.

Like the other national parks, this one has a junior ranger program. You complete a few things in an activity book and take a pledge to protect the park, and then you get a badge and some stickers. Here’s Oliver proudly taking his pledge, holding the new raccoon he bought in their gift shop.

Canyon, TX

Laura and I have been to Canyon, TX, twice — the first was for our honeymoon. Yes, we did get some strange looks when we told people we were going to Amarillo for our honeymoon. But it was absolutely perfect for us. We both enjoy the simple gifts of nature.

We kept thinking “we’ve got to take the boys here”. So this weekend, we did. We flew a Cessna out there.

Almost every little general aviation airport seems to have a bowl of candy, a plate of cookies, or some such thing for people that are flying through. I often let Jacob and Oliver choose ONE item.

They hit the jackpot when we stopped at West Woodward, Oklahoma for fuel and a break. Two whole fridges stocked with stuff: cans of pop in one, and all sorts of snacks in the other. In typical GA fashion, there was a jar in the fridge asking for $1 if you took something. And it clearly hadn’t been emptied in awhile.

They also had a nice lounge and a patio. Perfect for munching while watching the activities on the ramp.

After landing at the beautiful little Tradewind Airport in Amarillo, we ate dinner at Feldman’s Wrong-Way Diner in Canyon, TX. Oh my, was that ever popular with the boys.

They had a giant bin of crayons by the entrance. Jacob and Oliver each grabbed a fistful, and decided it would be fun to do some math problems while we wait. Oliver particularly got into that, and was quite accurate on his large addition problems. Impressive for a first-grader!

Of course, the big highlight of the area is Palo Duro Canyon. Jacob and Oliver were so eager to explore the canyon that they were just about bubbling over with excitement the night before. They decided that we should explore one of the most difficult trails in the canyon – one that would take us from the bottom of the canyon all the way to the top and back, about 2.5 miles each way.

And they LOVED it. We’d stop every few minutes to climb on some rocks, smash up some pieces of sandstone, munch on a snack, or even watch a lizard scurry past.

At the “trading post” in the canyon, both boys explored the gift shop. Jacob happily purchased a Texas magnet and Palo Duro Canyon keychain, which he carried around the rest of the weekend. Oliver loves stuffed animals, and he bought a cuddly little (but long) snake. When we got back to the hotel, he tied a couple of knots in it, and it became “snake airlines”. Here is the snake airline taking off.

He named it “Rattletail the friendly snake”, which I thought was a pretty nifty name.

Oliver wanted to help with the plane. He helped me tie it down in Amarillo, helped check it over during preflight, basically got involved in every part of it. Jacob studied aviation maps (sectionals) with me, planning our flight, figuring out how fast we’d go. I loaded Avare (an Android app) on an old tablet for him, so he had aviation maps in the cockpit just like me. He would be telling us how fast we were going every so often, pointing out landmarks, etc.

When it was time to head back home, both boys wanted to stay longer — a sure sign of a good trip. They wanted to hike another trail in the canyon, go back to the museum, and “eat at Feldman’s 18 more times.” (We got there twice, which was plenty for Laura and me!)

On our drive home, Oliver said, “Dad-o, will you teach me to be a pilot? You should be my flight instructor. Then I could fly everywhere with you.”

Now that just makes a dad’s day.

]]>http://changelog.complete.org/archives/9659-the-simple-joys-of-the-plains/feed49659Count me as a systemd converthttp://changelog.complete.org/archives/9655-count-me-as-a-systemd-convert
http://changelog.complete.org/archives/9655-count-me-as-a-systemd-convert#commentsThu, 21 Apr 2016 13:45:03 +0000http://changelog.complete.org/?p=9655Continue reading Count me as a systemd convert→]]>Back in 2014, I wrote about some negative first impressions of systemd. I also had a plea to debian-project to end all the flaming, pointing out that “jessie will still boot”, noting that my preference was for sysvinit but things are what they are and it wasn’t that big of a deal.

Although I still have serious misgivings about the systemd upstream’s attitude, I’ve got to say I find the system rather refreshing and useful in practice.

Here’s an example. I was debugging the boot on a server recently. It mounts a bunch of NFS filesystems and runs a third-party daemon that is started from an old-style /etc/init.d script.

We had a situation where the NFS filesystems the daemon required didn’t mount on boot. The daemon then was started, and unfortunately it basically does a mkdir -p on startup. So it started running and processing requests with negative results.

So there were two questions: why did the NFS filesystems fail to start, and how could we make sure the daemon wouldn’t start without them mounted? For the first, journalctl -xb was immensely helpful. It logged the status of each individual mount, and it turned out that it looked like a modprobe or kernel race condition when a bunch of NFS mounts were kicked off in parallel and all tried to load the nfsv4 module at the same time. That was easy enough to work around by adding nfsv4 to /etc/modules. Now for the other question: refusing to start the daemon if the filesystems weren’t there.

With systemd, this was actually trivial. I created /etc/systemd/system/mydaemon.service.requires (I’ll call the service “mydaemon” here), and in it I created a symlink to /lib/systemd/system/remote-fs.target. Then systemctl daemon-reload, and boom, done. systemctl list-dependencies mydaemon will even show the the dependency tree, color-coded status of each item on it, and will actually show every single filesystem that remote-fs requires and the status of it in one command. Super handy.

In a non-systemd environment, I’d probably be modifying the init script and doing a bunch of manual scripting to check the filesystems. Here, one symlink and one command did it, and I get tools to inspect the status of the mydaemon prerequisites for free.

I’ve got to say, as someone that has occasionally had to troubleshoot boot ordering and update-rc.d symlink hell, troubleshooting this stuff in systemd is considerably easier and the toolset is more powerful. Yes, it has its set of poorly-documented complexity, but then so did sysvinit.

I never thought the “world is falling” folks were right, but by now I can be counted among those that feels like systemd has matured to the point where it truly is superior to sysvinit. Yes, in 2014 it had some bugs, but by here in 2016 it looks pretty darn good and I feel like Debian’s decision has been validated through my actual experience with it.

]]>http://changelog.complete.org/archives/9655-count-me-as-a-systemd-convert/feed119655A Year of Flighthttp://changelog.complete.org/archives/9636-a-year-of-flight
http://changelog.complete.org/archives/9636-a-year-of-flight#commentsSat, 16 Apr 2016 02:36:56 +0000http://changelog.complete.org/?p=9636Continue reading A Year of Flight→]]>“Dad-o, I’m so glad you’re a pilot!”

My 9-year-old son Jacob has been saying that, always with a big hug and his fond nickname for me (“dad-o”). It has now been a year since the first time I sat in the pilot’s seat of a plane, taking my first step towards exploring the world from the sky. And now, one year after I first sat in the pilot’s seat of an airborne plane, it’s prompted me to think back to my own memories.

Flying over the airport at Moundridge, KS

Memories

Back when I was a child, maybe about the age my children are now, I’d be outside in the evening and see this orange plane flying overhead. Our neighbor Don had a small ultralight plane and a grass landing strip next to his house. I remember longing to be up in the sky with Don, exploring the world from up there. At that age, I didn’t know all the details of why that wouldn’t work — I just knew I wanted to ride in it.

It wasn’t until I was about 11 that I flew for the first time. I still remember that TWA flight with my grandma, taking off early in the morning and flying just a little ways above the puffy clouds lit up all yellow and orange by the sunrise. Even 25 years later, that memory still holds as one of the most beautiful scenes I have ever seen.

Exploring

I have always been an explorer.

When I go past something interesting, I love to go see what it looks like inside. I enjoy driving around Kansas with Laura, finding hidden waterfalls, old county courthouses, ghost towns, beautiful old churches, even small-town restaurants. I explore things around me, too — once taking apart a lawnmower engine as a child, nowadays building HF antennas in my treetops or writing code for Linux. If there is little to learn about something, it becomes less interesting to me.

I see this starting to build in my children, too. Since before they could walk, if we were waiting for something in a large building, we’d “go exploring.”

A patch of rain over Hillsboro, KS

The New World

A pilot once told me, “Nobody can become a pilot without it changing the way they see the world — and then, changing their life.”

I doubted that. But it was true. One of the most poetic sights I know is flying a couple thousand feet above an interstate highway at night, following it to my destination. All those red and white lights, those metal capsules of thousands of lives and thousands of stories, stretching out as far as the eye can see in either direction.

Kansas sunset from the plane

When you’re in a plane, that small town nowhere near a freeway that always seemed so far away suddenly is only a 15-minute flight away, not even enough time to climb up to a high cruise altitude. Two minutes after takeoff, any number of cities that are an hour’s drive away are visible simultaneously, their unique features already recognizable: a grain elevator, oil refinery, college campus, lake, whatever.

And all the houses you fly over — each with people in them. Some pretty similar to you, some apparently not. But pretty soon you realize that we all are humans, and we aren’t all that different. You can’t tell a liberal from a conservative from the sky, nor a person’s race or religion, nor even see the border between states. Towns and cities are often nameless from the sky, unless you’re really low; only your navigation will tell you where you are.

I’ve had the privilege to fly to small out-of-the-way airports, the kind that have a car that pilots can use for free to go into town and get lunch, and leave the key out for them. There I’ve met many friendly people. I’ve also landed my little Cessna at a big commercial airport where I probably used only 1/10th of the runway, on a grass runway that was barely maintained at all. I’ve flown to towns I’d driven to or through many times, discovering the friendly folks at the small airport out of town. I’ve flown to parts of Kansas I’ve never been to before, discovered charming old downtowns and rolling hills, little bursts of rain and beautiful sunsets that seem to turn into a sea.

For a guy that loves exploring the nooks and crannies of the world that everyone else drives by on their way to a major destination, being a pilot has meant many soul-filling moments.

Hard Work

I knew becoming a pilot would be a lot of hard work, and thankfully I remembered stories like that when I finally concluded it would be worth it. I found that I had an aptitude for a lot of things that many find difficult about being a pilot: my experience with amateur radio made me a natural at talking to ATC, my fascination with maps and navigation meant I already knew how to read aviation sectional maps before I even started my training and knew how to process that information in the cockpit, my years as a system administrator and programmer trained me with a careful and methodical decision-making process. And, much to the surprise of my flight instructor, I couldn’t wait to begin the part of training about navigating using VORs (VHF radio beacons). I guess he, like many student pilots, had struggled with that, but I was fascinated by this pre-GPS technology (which I still routinely use in my flight planning, as a backup in case the GPS constellation or a GPS receiver fails). So that left the reflexes of flight, the “art” of it, as the parts I had to work on the hardest.

The exam with the FAA is not like getting your driver’s license. It’s a multi-stage and difficult process. So when the FAA Designated Pilot Examiner said “congratulations, pilot!” and later told my flight instructor that “you did a really good job with this one,” I felt a true sense of accomplishment.

Some of my prep materials

Worth It

Passengers in a small plane can usually hear all the radio conversations going on. My family has heard me talking to air traffic control, to small and big planes. My 6-year-old son Oliver was playing yesterday, and I saw him pick up a plane and say this:

“Two-four-niner-golf requesting to land on runway one-seven…. Two-four-niner-golf back-taxi on one-seven… Two-four-niner-golf ready to takeoff on runway one-seven!”

That was a surprisingly accurate representation of some communication a pilot might have (right down to the made-up tailnumber with the spelling alphabet!)

It just got more involved from there!

Jacob and Oliver love model train shows. I couldn’t take them to one near us, but there was one in Joplin, MO. So the day before Easter, while Laura was working on her Easter sermon, two excited boys and I (frankly also excited) climbed into a plane and flew to Joplin.

We had a great time at the train show, discovered a restaurant specializing in various kinds of hot dogs (of course they both wanted to eat there), played in a park, explored the city, and they enjoyed the free cookies at the general aviation terminal building while I traded tips on fun places to fly with other pilots.

When it comes right down to it, the smiles of the people I fly with are the most beautiful thing in the air.

Jacob after his first father-son flight with me

]]>http://changelog.complete.org/archives/9636-a-year-of-flight/feed39636Free cars, sunsets, and Kansashttp://changelog.complete.org/archives/9630-free-cars-sunsets-and-kansas
http://changelog.complete.org/archives/9630-free-cars-sunsets-and-kansas#respondWed, 23 Mar 2016 22:44:20 +0000http://changelog.complete.org/?p=9630Continue reading Free cars, sunsets, and Kansas→]]>“Will you have a car I can borrow?” I asked.

“Sure. No charge. There’s a sign telling you where to find the key.”

It is pretty common for small airports to have a car for a pilot to borrow when flying in. This lets a person go into town for lunch, or visit friends. And it’s usually free, with a can to donate a few bucks or a polite request to fill up the tank when you’re done.

Still, when I had called ahead to ask about flying into the airport in a small town in north-central Kansas, I hadn’t expected to be told to just waltz into the place and take the key. But they had no staff at the airport most of the time. So, to me — another person from a small town — it made perfect sense. Somehow, because of that phone call, this town I had visited once, maybe 25 years ago, seemed instantly familiar.

My mom grew up in a small town near there. She wanted me to see where she grew up, to meet some people that meant a lot to her. As it’s quite a distance from home, I offered to fly her there. So, Laura, mom, and I climbed into a Cessna one morning for the flight northwest. We touched down at the airport, and I pulled the plane up to the little terminal building.

After I took care of parking the plane, I went to find the car. Except the car was missing. Some other pilot had flown in the same day and was using it, according to the logbook on the desk. I called the number on a sign — which rang to the sheriff’s office — and they confirmed it. According to the logbook, this was only the third time that car had been driven since Thanksgiving. Were we stuck at the airport a few miles out of down?

Nope. Mom called the people we were going to meet, a wonderful couple in their upper 80s. They drove out to pick us up. I’m rather glad the car was gone, because I had such a great time visiting with them. Norris told me about the days when the state highways were gravel — how they’d have to re-blade them every few days due to all the traffic. I heard about what happened when the people in that community heard of some folks in Africa in need of car equipment — they modified a tractor to fit in a shipping container and shipped it to Africa, along with a lot of books, blankets, supplies, and anything else needed to fill up a huge shipping container. Sounds like something people around here would do.

We drove around a couple of the small towns. The town my mom grew up in has seen better days. Its schools closed years ago, the old hotel whose owner gave her piano lessons is condemned, and many houses have been lost. But the town lives on. A new community center was built a few years ago. The grain elevator is expanding. Every time a business on Main Street closes, the grocery store expands a little bit: it’s now a grocery store with a little hardware store and a little restaurant mixed in. “The mall”, as the locals jokingly call it. And, of course, two beautiful small churches still meet every Sunday. Here’s the one my mom attended as a child.

We drove past the marker at the geographic center of the contiguous United States. Norris saw some other visitors, rolled down his windows, and treated them — and us — to an unexpected story of the time thousands of people banded together to completely build a house in a single day, just down the road. Smiles all around.

So here I was, nearly 200 miles from home, in an unfamiliar town – but one where I could just feel the goodness. After spending a few hours with these people, I felt like they were old friends.

As I flew us home, I spotted one of my favorite Kansas sights: a beautiful sunset. From the plane, it almost looks like the land at the horizon turns blue like the ocean, and above it the last hint of sun paints the canvas-sky.

In this week of controversy, politics, and reports of violence, it reminds me that we all get the privilege of sharing this beautiful Earth. I didn’t ask anybody on that trip about their politics, religion, or opinions on any of the divisive issues of the day. Whether they agree with me on those things or not is irrelevant. I got to spend a day with good-hearted and delightful people, so I flew back with a smile.

]]>http://changelog.complete.org/archives/9630-free-cars-sunsets-and-kansas/feed09630Bach, Dot Matrix Printers, and Dinnerhttp://changelog.complete.org/archives/9621-bach-dot-matrix-printers-and-dinner
http://changelog.complete.org/archives/9621-bach-dot-matrix-printers-and-dinner#respondWed, 27 Jan 2016 14:50:55 +0000http://changelog.complete.org/?p=9621Continue reading Bach, Dot Matrix Printers, and Dinner→]]>Dinner last night started out all normal. Then Jacob and Oliver started asking me about printers. First they wanted to know how an ink jet printer works. Then they wanted to know how a laser printer works. Then they wanted to know what would happen if you’d put ink in a laser printer or toner in an ink jet. They were fascinated as I described the various kinds of clogging and ruining that would inevitably occur.

So I looked up the misty details of dye sublimation and offset printing and described how they worked. That seemed to satisfy them. Then they asked me what my favorite kind of printer was. I said “dot matrix, because it makes the best sound.” That had their attention. They stopped eating to ask the vitally important question: “Dad, what sound does it make?” At this point, I did my best dot matrix impression at the dinner table, to much laughter and delight.

I guess it must have all sunk in, because this morning before school Jacob all of a sudden begged to see the fuser in my laser printer. So we turned it around, opened up the back panel — to his obvious excitement — and then I pointed to the fuser, with its “hot” label. I even heard a breathy “wow” from him.

]]>http://changelog.complete.org/archives/9621-bach-dot-matrix-printers-and-dinner/feed09621Hiking a mountain with Ian Murdockhttp://changelog.complete.org/archives/9437-hiking-a-mountain-with-ian-murdock
http://changelog.complete.org/archives/9437-hiking-a-mountain-with-ian-murdock#commentsMon, 04 Jan 2016 01:15:32 +0000http://changelog.complete.org/?p=9437Continue reading Hiking a mountain with Ian Murdock→]]>“Would you like to hike a mountain?” That question caught me by surprise. It was early in 2000, and I had flown to Tucson for a job interview. Ian Murdock was starting a new company, Progeny, and I was being interviewed for their first hire.

“Well,” I thought, “hiking will be fun.” So we rode a bus or something to the top of the mountain and then hiked down. Our hike was full of — well, everything. Ian talked about Tucson and the mountains, about his time as the Debian project leader, about his college days. I asked about the plants and such we were walking past. We talked about the plans for Progeny, my background, how I might fit in. It was part interview, part hike, part two geeks chatting. Ian had no HR telling him “you can’t go hiking down a mountain with a job candidate,” as I’m sure HR would have. And I am glad of it, because even 16 years later, that is still by far the best time I ever had at a job interview, despite the fact that it ruined the only pair of shoes I had brought along — I had foolishly brought dress shoes for a, well, job interview.

I guess it worked, too, because I was hired. Ian wanted to start up the company in Indianapolis, so over the next little while there was the busy work of moving myself and setting up an office. I remember those early days – Ian and I went computer shopping at a local shop more than once to get the first workstations and servers for the company. Somehow he had found a deal on some office space in a high-rent office building. I still remember the puzzlement on the faces of accountants and lawyers dressed up in suits riding in the elevators with us in our shorts and sandals, or tie-die, next to them.

Progeny’s story was to be a complicated one. We set out to rock the world. We didn’t. We didn’t set out to make lasting friendships, but we often did. We set out to accomplish great things, and we did some of that, too.

We experienced a full range of emotions there — elation when we got hardware auto-detection working well or when our downloads looked very popular, despair when our funding didn’t come through as we had hoped, being lost when our strategy had to change multiple times. And, as is the case everywhere, none of us were perfect.

I still remember the excitement after we published our first release on the Internet. Our little server that could got pegged at 100Mb of outbound bandwidth (that was something for a small company in those days.) The moment must have meant something, because I still have the mrtg chart from that day on my computer, 15 years later.

We made a good Linux distribution, an excellent Debian derivative, but commercial success did not flow from it. In the succeeding months, Ian and the company tried hard to find a strategy that would stick and make our big break. But that never happened. We had several rounds of layoffs when hoped-for funding never materialized. Ian eventually lost control of the company, and despite a few years of Itanium contract work after I left, closed for good.

Looking back, Progeny was life — compressed. During the good times, we had joy, sense of accomplishment, a sense of purpose at doing something well that was worth doing. I had what was my dream job back then: working on Debian as I loved to do, making the world a better place through Free Software, and getting paid to do it. And during the bad times, different people at Progeny experienced anger, cynicism, apathy, sorrow for the loss of our friends or plans, or simply a feeling to soldier on. All of the emotions, good or bad, were warranted in their own way.

Bruce Byfield, one of my co-workers at Progeny, recently wrote a wonderful memoriam of Ian. He wrote, “More than anything, he wanted to repeat his accomplishment with Debian, and, naturally he wondered if he could live up to his own expectations of himself. That, I think, was Ian’s personal tragedy — that he had succeeded early in life, and nothing else he did with his life could quite measure up to his expectations and memories.”

Ian was not the only one to have some guilt over Progeny. I, for years, wondered if I should have done more for the company, could have saved things by doing something more, or different. But I always came back to the conclusion I had at the time: that there was nothing I could do — a terribly sad realization.

In the years since, I watched Ubuntu take the mantle of easy-to-install Debian derivative. I saw them reprise some of the ideas we had, and even some of our mistakes. But by that time, Progeny was so thoroughly forgotten that I doubt they even realized they were doing it.

I had long looked at our work at Progeny as a failure. Our main goal was never accomplished, our big product never sold many copies, our company eventually shuttered, our rock-the-world plan crumpled and forgotten. And by those traditional measurements, you could say it was a failure.

But I have come to learn in the years since that success is a lot more that those things. Success is also about finding meaning and purpose through our work. As a programmer, success is nailing that algorithm that lets the application scale 10x more than before, or solving that difficult problem. As a manager, success is helping team members thrive, watching pieces come together on projects that no one person could ever do themselves. And as a person, success comes from learning from our experiences, and especially our mistakes. As J. Michael Straczynski wrote in a Babylon 5 episode, loosely paraphrased: “Maybe this experience will be a good lesson. Too bad it was so painful, but there ain’t no other kind.”

The thing about Progeny is this – Ian built a group of people that wanted to change the world for the better. We gave it our all. And there’s nothing wrong with that.

Progeny did change the world. As us Progeny alumni have scattered around the country, we benefit from the lessons we learned there. And many of us were “different”, sort of out of place before Progeny, and there we found others that loved C compilers, bootloaders, and GPL licenses just as much as we did. We belonged, not just online but in life, and we went on to pull confidence and skill out of our experience at Progeny and use them in all sorts of ways over the years.

And so did Ian. Who could have imagined the founder of Debian and Progeny would one day lead the cause of an old-guard Unix turning Open Source? I run ZFS on my Debian system today, and Ian is partly responsible for that — and his time at Progeny is too.

So I can remember Ian, and Progeny, as a success. And I leave you with a photo of my best memento from the time there: an original unopened boxed copy of Progeny Linux.

]]>http://changelog.complete.org/archives/9437-hiking-a-mountain-with-ian-murdock/feed89437Amtrak Airlineshttp://changelog.complete.org/archives/9433-amtrak-airlines
http://changelog.complete.org/archives/9433-amtrak-airlines#commentsTue, 22 Dec 2015 21:14:02 +0000http://changelog.complete.org/?p=9433Continue reading Amtrak Airlines→]]>I came downstairs this morning and found a surprise waiting for me. Chairs from all over had been gathered up and arranged in rows, airline style. Taped to the wall was a “food court” sign. At the front was a picture of an airplane, decked out with the Amtrak logo of all things, and a timetable taped to our dining room table.

Jacob soon got out string to be seatbelts, too. And, using his copy machine, printed out a picture of a wing to tape to the side of the “airplane”.

And here is the “food court” sign Oliver made:

This plane was, according to the boys, scheduled to leave at 9:30. It left a fashionable 2 hours late or so. They told me I would be the pilot, and had me find headphones to be my “headset”. (I didn’t wear my real headset on the grounds that then I wouldn’t be able to hear them.) Jacob decided he would be a flight attendant, his grandma would be the co-pilot, and Oliver would be the food court worker. The food court somehow seemed to travel with the plane.

Oliver made up a menu for the food court. It consisted of, and I quote: “trail mix, banana, trail mix, half banana, trail mix, trail mix, trail mix”. He’s already got the limited selection of airport food down pat, I can see.

Jacob said the flight would be from Chicago to Los Angeles, and so it was. Since it was Amtrak Airlines, we were supposed to pretend to fly over the train tracks the whole way.

If it’s not Christmas yet, we just invent some fun, eh? Pretty clever.

]]>http://changelog.complete.org/archives/9433-amtrak-airlines/feed49433Where does a person have online discussions anymore?http://changelog.complete.org/archives/9428-where-does-a-person-have-online-discussions-anymore
http://changelog.complete.org/archives/9428-where-does-a-person-have-online-discussions-anymore#commentsTue, 01 Dec 2015 00:22:11 +0000http://changelog.complete.org/?p=9428Continue reading Where does a person have online discussions anymore?→]]>Back in the day, way back in the day perhaps, there were interesting places to hang out online. FidoNet provided some discussion groups — some local, some more national or international. Then there was Usenet, with the same but on a more grand scale.

There were things I liked about both of them.

They fostered long-form, and long-term, discussion. Replies could be thoughtful, and a person could think about it for a day before replying.

Socially, you would actually get to know the people in the communities you participated in. There would be regulars, and on FidoNet at least, you might bump into them in different groups or even in real life. There was a sense of community. Moreover, there was a slight barrier to entry and that was, perhaps, a good thing; there were quite a lot of really interesting people and not so many people that just wanted answers to homework questions.

Technologically, you got to bring your own client. They were also decentralized, without any one single point of failure, and could be downloaded and used offline. You needed very little in terms of Internet connection.

They both had some downsides; Usenet, in particular, often lacked effective moderation. Not everyone wrote thoughtful posts.

Is there anything like it these days? I’ve sometimes heard people suggest Reddit. It shares some of those aspects, and even has some clients capable of offline operation. However, what it doesn’t really have is long-form discussion. I often find that if I am 6 hours late to a thread, nobody will bother to read my reply because it’s off their radar already. This happens so often that I rarely bother to participate anymore; I am not going to sit at reddit hitting refresh all day long.

There are a few web forums, but they suffer from all sorts of myriad problems; no cohesive community, the “hot topic” vanishing issue of Reddit, the single point of failure, etc.

For awhile, Google+ looked like it might head this way. But I don’t think it really has. I still feel as if there is a vacuum out there.

Any thoughts?

]]>http://changelog.complete.org/archives/9428-where-does-a-person-have-online-discussions-anymore/feed239428I do not fearhttp://changelog.complete.org/archives/9422-i-do-not-fear
http://changelog.complete.org/archives/9422-i-do-not-fear#commentsFri, 20 Nov 2015 19:22:15 +0000http://changelog.complete.org/?p=9422Continue reading I do not fear→]]>I am so saddened by the news this week. The attacks in Paris, Beirut, and Mali. The reaction of fear, anger, and hate. Governors racing to claim they will keep out refugees, even though they lack the power to do so. Congress voting to keep out refugees.

Emotions are a powerful thing. They can cause people to rise up and accomplish stunning things that move humanity forward. And they can move us back. Fear, and the manipulation of it, is one of those.

What have I to fear?

Even if the United States accepted half a million Syrian refugees tomorrow, I would be far more likely to die in a car accident than at the hands of a Syrian terrorist. I am a careful and cautious person, but I understand that life is not lived unless risk is balanced. I know there is a risk of being in a car crash every time I drive somewhere — but if that kept me at home, I would never see my kids’ violin concert, the beautiful “painted” canyon of Texas, or the Flint Hills of Kansas. So I drive smart and carefully, but I still drive without fear. I accept this level of risk as necessary to have a life worth living in this area (where there are no public transit options and the nearest town is miles away).

I have had pain in my life. I’ve seen grandparents pass away, I’ve seen others with health scares. These things are hard to think about, but they happen to us all at some point.

What have I to fear?

I do not fear giving food to the hungry, shelter to the homeless, comfort to those that have spent the last years being shot at. I do not fear helping someone that is different than me. If I fail to do these things for someone because of where they come from or what their holy book is, then I have become less human. I have become consumed by fear. I have let the terrorists have control over my life. And I refuse to do that.

If governors really wanted to save lives, they would support meaningful mass transit alternatives that would prevent tens of thousands of road deaths a year. They would support guaranteed health care for all. They would support good education, science-based climate change action, clean water and air, mental health services for all, and above all, compassion for everyone.

By supporting Muslim registries, we look like Hitler to them. By discriminating against refugees based on where they’re from or their religion, we support the terrorists, making it easy for them to win hearts and minds. By ignoring the fact that entering the country as a refugee takes years, as opposed to entering as a tourist taking only minutes, we willfully ignore the truth about where dangers lie.

So what do I have to fear?

Only, as the saying goes, fear. Fear is making this country turn its backs on the needy. Fear is making not just the US but much of Europe turn its backs on civil liberties and due process. Fear gives the terrorists control, and that helps them win.

I refuse. I simply refuse to play along. No terrorist, no politician, no bigot gets to steal MY humanity.

Ultimately, however, I know that the long game is not one of fear. The arc of the universe bends towards justice, and ultimately, love wins. It takes agonizingly long sometimes, but in the end, love wins.

So I do not fear.

]]>http://changelog.complete.org/archives/9422-i-do-not-fear/feed109422Memories of a printerhttp://changelog.complete.org/archives/9420-memories-of-a-printer
http://changelog.complete.org/archives/9420-memories-of-a-printer#commentsFri, 13 Nov 2015 18:42:58 +0000http://changelog.complete.org/?p=9420Continue reading Memories of a printer→]]>I have a friend who hates printers. I’ll call him “Mark”, because that, incidentally, is his name. His hatred for printers is partly my fault, but that is, ahem, a story for another time that involves him returning from a battle with a printer with a combination of weld dust, toner, and a deep scowl on his face.

I also tend to hate printers. Driver issues, crinkled paper, toner spilling all over the place…. everybody hates printers.

But there is exactly one printer that I have never hated. It’s almost 20 years old, and has some stories to tell.

Nearly 20 years ago, I was about to move out of my parents’ house, and I needed a printer. I bought a LaserJet 6MP. This printer ought to have been made by Nokia. It’s still running fine, 18 years later. It turned out to be one of the best investments in computing equipment I’ve ever made. Its operating costs, by now, are cheaper than just about any printer you can buy today — less than one cent per page. It has been supported by every major operating system for years.

PostScript was important, because back then running Ghostscript to convert to PCL was both slow and a little error-prone. PostScript meant I didn’t need a finicky lpr/lprng driver on my Linux workstation to print. It just… printed. (Hat tip to anyone else that remembers the trial and error of constructing an /etc/printcap that would print both ASCII and PostScript files correctly!)

Out of this printer have come plane and train tickets, taking me across the country to visit family and across the world to visit friends. It’s printed resumes and recipes, music and university papers. I even printed wedding invitations and envelopes on them two years ago, painstakingly typeset in LaTeX and TeXmacs. I remember standing at the printer in the basement one evening, feeding envelope after envelope into the manual feed slot. (OK, so it did choke on a couple of envelopes, but overall it all worked great.)

The problem, though, is that it needs a parallel port. I haven’t had a PC with one of those in a long while. A few years ago, in a moment of foresight, I bought a little converter box that has an Ethernet port and a parallel port, with the idea that it would be pay for itself by letting me not maintain some old PC just to print. Well, it did, but now the converter box is dying! And they don’t make them anymore. So I finally threw in the towel and bought a new LaserJet.

It cost a third of what the 6MP did, has a copier, scanner, prints in color, does duplexing, has wifi… and, yes, still supports PostScript — strangely enough, a deciding factor in going with HP over Brother once again. (The other was image quality)

We shall see if I am still using it when I’m 50.

]]>http://changelog.complete.org/archives/9420-memories-of-a-printer/feed119420Wow. I did that!http://changelog.complete.org/archives/9407-wow-i-did-that
http://changelog.complete.org/archives/9407-wow-i-did-that#commentsTue, 10 Nov 2015 19:19:52 +0000http://changelog.complete.org/?p=9407Continue reading Wow. I did that!→]]>It’s now official: I’m a pilot. This has been one of the most challenging, and also most rewarding, journeys I’ve been on. It had its moments of struggle, moments of joy, moments of poetry. I wrote about the poetry of flying at night recently. Here is the story of my first landing on a grass runway, a few months ago.

Grass

Where the air is so pure, the zephyrs so free,
The breezes so balmy and light,
That I would not exchange my home on the range
For all of the cities so bright.

– John A. Lomax

We are used to seeing planes in these massive palaces of infrastructure we call airports. We have huge parking garages, giant terminals, security lines hundreds of people deep, baggage carts, jetways, video screens, restaurants, and miles and miles of concrete.

But most of the world’s airports are not like that. A pilot of a small plane gets to see the big airports, sure, but we also get to use the smaller airports — hidden in plain sight to most.

Have you ever taken off from a strip of grass? As I told my flight instructor when I tried it for the first time, “I know this will work, but somehow I will still be amazed if it actually does.”

I took off from a strip of grass not long ago. The airport there had one paved runway, and the rest were grass. Short runways, narrow runways, grass runways. No lights. No paint. No signs. No trucks, no jetways, nothing massive. In fact, no people. Just a mowed path and a couple of yellow or white markers.

I taxied down the grass runway, being careful to never let the plane’s wheels stop moving lest the nose gear get stuck in a pothole. I felt all the bumps in the ground as we moved.

End of runway. Turn the plane around. A little bit of flap for more lift, full throttle, mind the centerline — imaginary centerline, this time. It starts picking up speed, slower than usual, bump bump bump. Those buildings at the end of the runway are staring me down. More speed, and suddenly the runway feels smooth; it has enough lift to keep from falling into every bump. Then we lift off just a touch; I carefully keep the plane down until we pick up enough speed to ascend farther, then up we go. I keep a watchful eye on those buildings straight ahead and that water tower just slightly off to the one side. We climb over a lake as I watch that water tower pass safely below and to the side of the plane. It had worked, and I had a smile of amazement.

With a half mile of grass, you really can go anywhere.

Many times I had driven within half a mile of that runway, but never seen it. Never wondered where people go after using it. Never realizing that, although it’s a 45-minute drive from my house, it’s really pretty close. Never understanding that “where people go” after taking off from that runway is “everywhere”.

]]>http://changelog.complete.org/archives/9407-wow-i-did-that/feed69407The Train to Galesburghttp://changelog.complete.org/archives/9413-the-train-to-galesburg
http://changelog.complete.org/archives/9413-the-train-to-galesburg#commentsWed, 28 Oct 2015 01:13:32 +0000http://changelog.complete.org/?p=9413Continue reading The Train to Galesburg→]]>Sometimes, children are so excited you just can’t resist. Jacob and Oliver have been begging for a train trip for awhile now, so Laura and I took advantage of a day off school to take them to the little town of Galesburg, IL for a couple days.

There was excitement in the air. I was asked to supply a bedtime story about trains — I did. On the way to the train station — in the middle of the night — there was excited jabbering about trains. Even when I woke them up, they lept out of bed and raced downstairs, saying, “Dad, why aren’t you ready yet?”

As the train was passing through here at around 4:45AM, and we left home with some time to spare, we did our usual train trip thing of stopping at the one place open at such a time: Druber’s Donuts.

Much as Laura and I might have enjoyed some sleep once we got on the train, Jacob and Oliver weren’t having it. Way too much excitement was in the air. Jacob had his face pressed against the window much of the time, while Oliver was busy making “snake trains” from colored clay — complete with eyes.

The boys were dressed up in their train hats and engineer overalls, and Jacob kept musing about what would happen if somebody got confused and thought that he was the real engineer. When an Amtrak employee played along with that later, he was quite thrilled!

We were late enough into Galesburg that we ate lunch in the dining car. A second meal there — what fun! Here they are anxiously awaiting the announcement that the noon reservations could make their way to the dining car. Oh, and jockeying for position to see who would be first and get to do the all-important job of pushing the button to open the doors between train cars.

Even waiting for your food can be fun.

Upon arriving, we certainly couldn’t leave the train station until our train did, even though it was raining.

Right next to the train station is the Discovery Depot Children’s Museum. It was a perfect way to spend a few hours. Jacob really enjoyed the building wall, where you can assemble systems that use gravity (really a kinetic/potential energy experiment wall) to funnel rubber balls all over the place. He sticks out his tongue when he’s really thinking. Fun to watch.

Meanwhile, Oliver had a great time with the air-powered tube system, complete with several valves that can launch things through a complicated maze of transparent tubes.

They both enjoyed pretending I was injured and giving me rides in the ambulance. I was diagnosed with all sorts of maladies — a broken leg, broken nose. One time Jacob held up the pretend stethoscope to me, and I said “ribbit.” He said, “Dad, you’ve got a bad case of frog! You will be in the hospital 190 days!” Later I would make up things like “I think my gezotnix is all froibled” and I was ordered to never leave the ambulance again. He told the story of this several times.

After the museum closed, we ate supper. Keep in mind the boys had been up since the middle of the night without sleeping and were still doing quite well! They did start to look a bit drowsy — I thought Oliver was about to fall asleep, but then their food came. And at the hotel, they were perfectly happy to invent games involving jumping off the bed.

Saturday, we rode over to Peck Park. We had heard about this park from members of our church in Kansas, but oddly even the taxi drivers hadn’t ever heard of it. It’s well known as a good place to watch trains, as it has two active lines that cross each other at a rail bridge. And sure enough, in only a little while, we took in several trains.

The rest of that morning, we explored Galesburg. We visited an antique mall and museum, saw the square downtown, and checked out a few of the shops — my favorite was the Stray Cat, featuring sort of a storefront version of Etsy with people selling art from recycled objects. But that wasn’t really the boys’ thing, so we drifted out of there on our way to lunch at Baked, where we had some delicious deep-dish pizza.

After that, we still had some time to kill before getting back on the train. We discussed our options. And what do you know — we ended up back at the children’s museum. We stopped at a bakery to get the fixins for a light supper on the train, and ate a nice meal in the dining car once we got on. Then, this time, they actually slept.

Before long, it was 3AM again and time to get back off the train. Oliver was zonked out sleepy. Somehow I managed to get his coat and backpack on him despite him being totally limp, and carried him downstairs to get off the train. Pretty soon we walked to our car and drove home.

We tucked them in, and then finally tucked ourselves in. Sometimes being really tired is well worth it.

]]>http://changelog.complete.org/archives/9413-the-train-to-galesburg/feed19413Objects On Earth Are Closer Than They Appearhttp://changelog.complete.org/archives/9402-objects-on-earth-are-smaller-than-they-appear
http://changelog.complete.org/archives/9402-objects-on-earth-are-smaller-than-they-appear#commentsTue, 06 Oct 2015 23:39:42 +0000http://changelog.complete.org/?p=9402Continue reading Objects On Earth Are Closer Than They Appear→]]>“We all live beneath the great Big Dipper.”

So goes a line in a song I once heard the great Tony Brown sing. As I near the completion of my private pilot’s training, I’ve had more and more opportunities to literally see the wisdom in those words. Here’s a story of one of them.

Night

“A shining beacon in space — all alone in the night.”

– Babylon 5

A night cross-country flight, my first, taking off from a country airport. The plane lifts into the dark sky. The bright white lights of the runway get smaller, and disappear as I pass the edge of the airport. Directly below me, it looks like a dark sky; pitch black except for little pinpoints of light at farmhouses and the occasional car. But seconds later, an expanse of light unfolds, from a city it takes nearly an hour to reach by car. Already it is in sight, and as I look off to other directions, other cities even farther away are visible, too. The ground shows a square grid, the streets of the city visible for miles.

There are no highway signs in the sky. There are no wheels to keep my plane pointed straight. Even if I point the plane due south, if there is an east wind, I will actually be flying southwest. I use my eyes, enhanced by technology like a compass, GPS, and VHF radio beacons, to find my way. Before ever getting into the airplane, I have carefully planned my route, selecting both visual and technological waypoints along the way to provide many ways to ensure I am on course and make sure I don’t get lost.

Soon I see a flash repeating every few seconds in the distance — an airport beacon. Then another, and another. Little pinpoints of light nestled in the square orange grid. Wichita has many airports, each with its beacon, and one of them will be my first visual checkpoint of the night. I make a few clicks in the cockpit, and soon the radio-controlled lights at one of the airports spring to life, illuminating my first checkpoint. More than a mile of white lights there to welcome any plane that lands, and to show a point on the path of any plane that passes.

I continue my flight, sometimes turning on lights at airports, other times pointing my plane at lights from antenna towers (that are thousands of feet below me), sometimes keeping a tiny needle on my panel centered on a radio beacon. I land at a tiny, deserted airport, and then a few minutes later at a large commercial airport.

On my way back home, I fly solely by reference to the ground — directly over a freeway. I have other tools at my disposal, but don’t need them; the steady stream of red and white lights beneath me are all I need.

From my plane, there is just red and white. One after another, passing beneath me as I fly over them at 115 MPH. There is no citizen or undocumented immigrant, no rich or poor, no atheist or Christian or Muslim, no Democrat or Replubican, no American or Mexican, no adult or child, no rich or poor, no Porsche or Kia. Just red and white points of light, each one the same as the one before and the one after, stretching as far as I can see into the distance. All alike in the night.

You only need to get a hundred feet off the ground before you realize how little state lines, national borders, and the machinery of politics and exclusion really mean. From the sky, the difference between a field of corn and a field of wheat is far more significant than the difference between Kansas and Missouri.

This should be a comforting reminder to us. We are all unique, and beautiful in our uniqueness, but we are all human, each as valuable as the next.

Up in the sky, even though my instructor was with me, during quiet times it is easy to feel all alone in the night. But I know it is not the case. Only a few thousand feet separate my plane from those cars. My plane, too, has red and white lights.

How often at night, when the heavens were bright,
With the light of the twinkling stars
Have I stood here amazed, and asked as I gazed,
If their glory exceed that of ours.

– John A. Lomax

]]>http://changelog.complete.org/archives/9402-objects-on-earth-are-smaller-than-they-appear/feed19402There’s still a chance to save WiFihttp://changelog.complete.org/archives/9396-theres-still-a-chance-to-save-wifi
http://changelog.complete.org/archives/9396-theres-still-a-chance-to-save-wifi#commentsThu, 03 Sep 2015 19:12:48 +0000http://changelog.complete.org/?p=9396Continue reading There’s still a chance to save WiFi→]]>You may not know it, but wifi is under assault in the USA due to proposed FCC regulations about modifications to devices with modular radios. In short, it would make it illegal for vendors to sell devices with firmware that users can replace. This is of concern to everyone, because Wifi routers are notoriously buggy and insecure. It is also of special concern to amateur radio hobbyists, due to the use of these devices in the Amateur Radio Service (FCC Part 97).

I submitted a comment to the FCC about this, which I am pasting in here. This provides a background and summary of the issues for those that are interested. Here it is:

My comment has two parts: one, the impact on the Amateur Radio service; and two, the impact on security. Both pertain primarily to the 802.11 (“Wifi”) services typically operating under Part 15.

The Amateur Radio Service (FCC part 97) has long been recognized by the FCC and Congress as important to the nation. Through it, amateurs contribute to scientific knowledge, learn skills that bolster the technological competitiveness of the United States, and save lives through their extensive involvement in disaster response.

Certain segments of the 2.4GHz and 5GHz Wifi bands authorized under FCC Part 15 also fall under the frequencies available to licensed amateurs under FCC Part 97 [1].

By scrupulously following the Part 97 regulations, many amateur radio operators are taking devices originally designed for Part 15 use and modifying them for legal use under the Part 97 Amateur Radio Service. Although the uses are varied, much effort is being devoted to fault-tolerant mesh networks that provide high-speed multimedia communications in response to a disaster, even without the presence of any traditional infrastructure or Internet backbone. One such effort [2] causes users to replace the firmware on off-the-shelf Part 15 Wifi devices, reconfiguring them for proper authorized use under Part 97. This project has many vital properties, particularly the self-discovery of routes between nodes and self-healing nature of the mesh network. These features are not typically available in the firmware of normal Part 15 devices.

It should also be noted that there is presently no vendor of Wifi devices that operate under Part 97 out of the box. The only route available to amateurs is to take Part 15 devices and modify them for Part 97 use.

Amateur radio users of these services have been working for years to make sure they do not cause interference to Part 15 users [3]. One such effort takes advantage of the modular radio features of consumer Wifi gear to enable communication on frequencies that are within the Part 97 allocation, but outside (though adjacent) to the Part 15 allocation. For instance, the chart at [1] identifies frequencies such as 2.397GHz or 5.660GHz that will never cause interference to Part 15 users because they lie entirely outside the Part 15 Wifi allocation.

If the FCC prevents the ability of consumers to modify the firmware of these devices, the following negative consequences will necessarily follow:

1) The use of high-speed multimedia or mesh networks in the Amateur Radio service will be sharply curtailed, relegated to only outdated hardware.

2) Interference between the Amateur Radio service — which may use higher power or antennas with higher gain — and Part 15 users will be expanded, because Amateur Radio service users will no longer be able to intentionally select frequencies that avoid Part 15.

3) The culture of inventiveness surrounding wireless communication will be curtailed in an important way.

Besides the impact on the Part 97 Amateur Radio Service, I also wish to comment on the impact to end-user security. There have been a terrible slew of high-profile situations where very popular consumer Wifi devices have had incredible security holes. Vendors have often been unwilling to address these issues [4].

Michael Horowitz maintains a website tracking security bugs in consumer wifi routers [5]. Sadly these bugs are both severe and commonplace. Within just the last month, various popular routers have been found vulnerable to remote hacking [6] and platforms for launching Distributed Denial-of-Service (DDoS) attacks [7]. These impacted multiple models from multiple vendors. To make matters worse, most of these issues should have never happened in the first place, and were largely the result of carelessness or cluelessness on the part of manufacturers.

Consumers should not be at the mercy of vendors to fix their own networks, nor should they be required to trust unauditable systems. There are many well-regarded efforts to provide better firmware for Wifi devices, which still keep them operating under Part 15 restrictions. One is OpenWRT [8], which supports a wide variety of devices with a system built upon a solid Linux base.

Please keep control of our devices in the hands of consumers and amateurs, for the benefit of all.

]]>http://changelog.complete.org/archives/9396-theres-still-a-chance-to-save-wifi/feed29396First steps: Debian on an Asus t100, and some negative experience with Gnomehttp://changelog.complete.org/archives/9394-first-steps-debian-on-an-asus-t100-and-some-negative-experience-with-gnome
http://changelog.complete.org/archives/9394-first-steps-debian-on-an-asus-t100-and-some-negative-experience-with-gnome#commentsSun, 16 Aug 2015 21:25:13 +0000http://changelog.complete.org/?p=9394Continue reading First steps: Debian on an Asus t100, and some negative experience with Gnome→]]>The Asus t100 tablet is this amazing and odd little thing: it sells for under $200, yet has a full-featured Atom 64-bit CPU, 2GB RAM, 32 or 64GB SSD, etc. By default, it ships with Windows 8.1. It has a detachable keyboard, so it can be used as a tablet or a very small 10″ laptop.

I have never been a fan of Windows on it. It does the trick for web browsing and email, but I’d like to ssh into my machines sometimes, and I just can’t bring myself to type sensitive passwords into Windows.

I decided to try installing Debian on it. After a lot of abortive starts due to the UEFI-only firmware, I got jessie installed. (The installer was fine; it was Debian Live that wouldn’t boot.) I got wifi and battery status working via an upgrade to the 4.1 kernel. A little $10 Edimax USB adapter was handy to spare a bunch of copying via USB disks.

I have been using XFCE with XMonad for so many years that I am somewhat a stranger to other desktop environments. XMonad isn’t really suitable for a tablet, however, so I thought I’d try Gnome, especially after a fairly glowing review about its use on a tablet.

I am already disappointed after just a few minutes. There is no suspend button on the menu. Some Googling showed that holding Alt while hovering over the power off button will change it to a suspend button. And indeed it does. But… uh, what? That is so common and so non-obvious. And pushing the power button does… nothing. That’s right, nothing. Apparently the way to enable some action when you push the power button is to type in a settings command in a terminal. There’s no setting in the settings panel.

I initially ditched Gnome some years ago due to its penchant for removing features. I had hoped that this much time later, it would have passed that stage, but I’m already disappointed. I was hoping for some really nice integration with the system. But my XFCE setup has a very clear “When power button is pressed” setting. I have no idea why Gnome doesn’t.

Also, the touch screen works fine and it registers my touches, but whenever I touch anywhere, the cursor disappears. Weird, eh?

There are some things to fix yet on the tablet (sound, brightness adjustment, and making suspend reliable) but others have solved these in Ubuntu so I don’t think it’ll be too hard.

In the meantime, any suggestions regarding Gnome? Is it just going to annoy me? Maybe I should try KDE also. I’ve heard good things about Plasma Active, but don’t see it in Debian though.

This post is already huge, so I am not going to dive into — much — specific commands, but I am linking to many sources with detailed instructions.

I’ve reviewed several types of cards. For this review, I will focus on the OpenPGP card and the Yubikey NEO, since the Cardomatic Smartcard-HSM is not supported by the gpg version in Jessie.

Both cards are produced by people with strong support for the Free Software ecosystem and have strong cross-platform support with source code.

OpenPGP card: Basics with GnuPG

The OpenPGP card is well-known as one of the first smart cards to work well on Linux. It is a single-application card focused on use with GPG. Generally speaking, by the way, you want GPG2 for use with smartcards.

Basically, this card contains three slots: decryption, signing, and authentication slots. The concept is that the private key portions of the keys used for these items are stored only on the card, can never be extracted from the card, and the cryptographic operations are performed on the card. There is more information in my original post. In a fairly rare move for smartcards, this card supports 4096-byte RSA keys; most are restricted to 2048-byte keys.

As you’ll see in a bit, most of this information also pertains to the OpenPGP mode of the Yubikey Neo.

OpenPGP card: Other uses

Of course, this is already pretty great to enhance your GPG security, but there’s a lot more that you can do with this card to add two-factor authentication (2FA) to a lot of other areas. Here are some pointers:

Now, however, the OpenSC project now supports the OpenPGP card as a PKCS#11 and PKCS#15 card, which means it works natively with ssh-agent as well. Try just ssh-add -s /usr/lib/x86_64-linux-gnu/pkcs11/opensc-pkcs11.so if you’ve put a key in the auth slot with GPG. ssh-add -L will list its fingerprint for insertion into authorized_keys. Very simple!

As an aside: Comments that you need scute for PKCS#11 support are now outdated. I do not recommend scute. It is quite buggy.

Between the use with ssh and the use with PAM, we have now covered 2FA for both local and remote use in Unix environments.

OpenPGP card: use on Windows

Let’s move on to Windows environments. The standard suggestion here seems to be the mysmartlogon OpenPGP mini-driver. It works with some sort of Windows CA system, or the local accounts using EIDAuthenticate. I have not yet tried this.

OpenPGP card: Use with X.509 or Windows Active Directory

You can use the card in X.509 mode via these gpgsm instructions, which apparently also work with Windows Active Directory in some fashion.

You can also use it with web browsers to present a certificate from a client for client authentication. For example, here are OpenSC instructions for Firefox.

You’ll want to install the opensc-pkcs11 package, and then give the path /usr/lib/x86_64-linux-gnu/pkcs11/opensc-pkcs11.so whenever something needs the PKCS#11 library. There seem to be some locking/contention issues between GPG2 and OpenSC, however. Usually killing pcscd and scdaemon will resolve this.

I would recommend doing manipulation operations (setting PINs, generating or uploading keys, etc.) via GPG2 only. Use the PKCS#11 tools only to access.

OATH TOTP, with a mobile or desktop helper app (drop-in for Google Authenticator

OATH HOTP

Challenge-response mode

U2F (Universal 2nd Factor) with Chrome

There is a ton to digest with this device.

Yubikey Neo Basics

By default, the Yubikey Neo is locked to only a subset of its features. Using the yubikey-personalization tool (you’ll need the version in stretch; jessie is too old), you can use ykpersonalize -m86 to unlock the full possibilities of the card. Run that command, then unplug and replug the device.

It will present itself as a USB keyboard as well as a PC/SC-compatible card reader. It has a capacitive button, which is used to have it generate keystrokes to input validation information for HOTP or HMAC validation. It has two “slots” that can be configured with HMAC and HOTP; a short button press selects the default slot #1 and a long press selects slot #2.

But before we get into that, let’s step back at some basics.

opensc-tool –list-algorithms claims this card supports RSA with 1024, 2048, and 3072 sizes, and EC with 256 and 384-bit sizes. I haven’t personally verified anything other than RSA-2048 though.

Yubikey Neo: OpenPGP support

In this mode, the card is mostly compatible with the physical OpenPGP card. I say “mostly” because there are a few protocol differences I’ll get into later. It is also limited to 2048-byte keys.

Support for this is built into GnuPG and the GnuPG features described above all work fine.

You can use gnupg-agent to use the card with SSH as before. However, due to some incompatibilities, the OpenPGP applet on this card cannot be used as a PKCS#11 card with either scute or OpenSC. That is not exactly a huge problem, however, as the card has another applet (PIV) that is compatible with OpenSC and so this still provides an avenue for SSH, OpenVPN, Mozilla, etc.

It should be noted that the OpenPGP applet on this card can also be used with NFC on Android with the OpenKeychain app. Together with pass (or its Windows, Mac, or phone ports), this makes a nicely secure system for storing passwords.

Yubikey Neo: PKCS#11 with the PIV applet

There is also support for the PIV standard on the Yubikey Neo. This is supported by default on Linux (via OpenSC) and Windows and provides a PKCS#11-compabible store. It should, therefore, be compatible with ssh-agent, OpenVPN, Active Directory, and all the other OpenPGP card features described above. The only difference is that it uses storage separate from the OpenPGP applet.

You will need one of the Yubico PIV tools to configure the key for it; in Debian, the yubico-piv-tool from stretch does this.

I couldn’t get this to work with Firefox, but it worked pretty much everywhere else.

Yubikey Neo: HOTP authentication

This is the default mode for your Yubikey; all other modes require enabling with ykpersonalize. In this mode, a 128-bit AES key stored on the Yubikey is used to generate one-time passwords (OTP). (This key was shared in advance with the authentication server.) A typical pattern would be for three prompts: username, password, and Yubikey HOTP. The user clicks in the Yubikey HOTP field, touches the Yubikey, and their one-time token is pasted in.

In the background, the service being authenticated to contacts an authentication server. This authentication server can be either your own (there are several open source implementations in Debian) or the free Yubicloud.

Either way, the server decrypts the encrypted part of the OTP, performs validity checks (making sure that the counter is larger than any counter it’s seen before, etc) and returns success or failure back to the service demanding authentication.

The first few characters of the posted auth contain the unencrypted key ID, and thus it can also be used to provide username if desired.

This mode is simple to implement, but it has a few downsides. One is that it is specific to the Yubico line of products, and thus has a vendor lock-in factor. Another is the dependence on the authentication server; this creates a potential single point of failure and can be undesireable in some circumtances.

Yubikey Neo: OATH and HOTP and TOTP

First, a quick note: OATH and OAuth are not the same. OATH is an authentication protocol, and OAuth is an authorization protocol. Now then…

Like Yubikey HOTP, OATH (both HOTP and TOTP) modes rely on a pre-shared key. (See details in the Yubico article.) Let’s talk about TOTP first. With TOTP, there is a pre-shared secret with each service. Each time you authenticate to that service, your TOTP generator combines the timestamp with the shared secret using a HMAC algorithm and produces a OTP that changes every 30 seconds. Google Authenticator is a common example of this protocol, and this is a drop-in replacement for it. Gandi has a nice description of it that includes links to software-only solutions on various platforms as well.

With the Yubikey, the shared secrets are stored on the card and processed within it. You cannot extract the shared secret from the Yubikey. Of course, if someone obtains physical access to your Yubikey they could use the shared secret stored on it, but there is no way they can steal the shared secret via software, even by compromising your PC or phone.

Since the Yubikey does not have a built-in clock, TOTP operations cannot be completed solely on the card. You can use a PC-based app or the Android application (Play store link) with NFC to store secrets on the device and generate your TOTP codes. Command-line users can also use the yubikey-totp tool in the python-yubico package.

OATH can also use HOTP. With HOTP, an authentication counter is used instead of a clock. This means that HOTP passwords can be generated entirely within the Yubikey. You can use ykpersonalize to configure either slot 1 or 2 for this mode, but one downside is that it can really only be used with one service per slot.

OATH support is all over the place; for instance, there’s libpam-oath from the OATH toolkit for Linux platforms. (Some more instructions on this exist.)

Note: There is another tool from Yubico (not in Debian) that can apparently store multiple TOTP and HOTP codes in the Yubikey, although ykpersonalize and other documentation cannot. It is therefore unclear to me if multiple HOTP codes are supported, and how..

Yubikey Neo: Challenge-Response Mode

This can be useful for doing offline authentication, and is similar to OATH-HOTP in a sense. There is a shared secret to start with, and the service trying to authenticate sends a challenge to the token, which must supply an appropriate response. This makes it only suitable for local authentication, but means it can be done fairly automatically and optionally does not even require a button press.

To muddy the waters a bit, it supports both “Yubikey OTP” and HMAC-SHA1 challenge-response modes. I do not really know the difference. However, it is worth noting that libpam-yubico works with HMAC-SHA1 mode. This makes it suitable, for instance, for logon passwords.

Yubikey Neo: U2F

U2F is a new protocol for web-based apps. Yubico has some information, but since it is only supported in Chrome, it is not of interest to me right now.

Do not think a hardware security token is a panacea. It is best used as part of a multi-factor authentication system; you don’t want a lost token itself to lead to a breach, just as you don’t want a compromised password due to a keylogger to lead to a breach.

These things won’t prevent someone that has compromised your PC from abusing your existing ssh session (or even from establishing new ssh sessions from your PC, once you’ve unlocked the token with the passphrase). What it will do is prevent them from stealing your ssh private key and using it on a different PC. It won’t prevent someone from obtaining a copy of things you decrypt on a PC using the Yubikey, but it will prevent them from decrypting other things that used that private key. Hopefully that makes sense.

One also has to consider the security of the hardware. On that point, I am pretty well satisfied with the Yubikey; large parts of it are open source, and they have put a lot of effort into hardening the hardware. It seems pretty much impervious to non-government actors, which is about the best guarantee a person can get about anything these days.

There is something about that moment. Hiking in the mountains near Durango, Colorado, with Laura and the boys, we found a beautiful spot with a view of the valley. We paused to admire, and then –

The sound of a steam locomotive whistle from down below, sounding loud all the way up there, then echoing back and forth through the valley. Then the quieter, seemingly more distant sound of the steam engine heading across the valley, chugging and clacking as it goes. More whistles, the sight of smoke and then of the train full of people, looking like a beautiful model train from our vantage point.

I’ve heard that sound on a few rare recordings, but never experienced it. I’ve been on steam trains a few times, but never spent time in a town where they still run all day, every day. It is a different sort of feeling to spend a week in a place where Jacob and Oliver would jump up several times a day and rush to the nearest window in an attempt to catch sight of the train.

Airplanes really can be a time machine in a sense — what a wondrous time to be alive, when things so ancient are within the reach of so many. I have been transported to Lübeck and felt the uneven 700-year-old stones of the Marienkirche underneath my feet, feeling a connection to the people that walked those floors for centuries. I felt the same in Prague, in St. George’s Basilica, built in 1142, and at the Acropolis of Lindos, with its ancient Greek temple ruins. In Kansas, I feel that when in the middle of the Flint Hills — rolling green hills underneath the pure blue sky with billowing white clouds, the sounds of crickets, frogs, and cicadas in my ears; the sights and sounds are pretty much as they’ve been for tens of thousands of years. And, of course, in Durango, arriving on a plane but seeing the steam train a few minutes later.

It was fitting that we were in Durango with Laura’s parents to celebrate their 50th anniversary. As we looked forward to riding the train, we heard their stories of visits to Durango years ago, of their memories of days when steam trains were common. We enjoyed thinking about what our lives would be like should we live long enough to celebrate 50 years of marriage. Perhaps we would still be in good enough health to be able to ride a steam train in Durango, telling about that time when we rode the train, which by then will have been pretty much the same for 183 years. Or perhaps we would take them to our creek, enjoying a meal at the campfire like I’ve done since I was a child.

Each time has its unique character. I am grateful for the cameras and airplanes and air conditioning we have today. But I am also thankful for those things that connect us with each other trough time, those rocks that are the same every year, those places that remind us how close we really are to those that came before.

]]>http://changelog.complete.org/archives/9368-the-time-machine-of-durango/feed09368True Things About Learning to Flyhttp://changelog.complete.org/archives/9364-true-things-about-learning-to-fly
http://changelog.complete.org/archives/9364-true-things-about-learning-to-fly#commentsSat, 18 Jul 2015 02:40:47 +0000http://changelog.complete.org/?p=9364Continue reading True Things About Learning to Fly→]]>I’ve been pretty quiet for the last few months because I’m learning to fly. I want to start with a few quotes about aviation. I have heard things like these from many people and can vouch for their accuracy:

Anyone can learn to fly.

Learning to fly is one of the hardest things you’ll ever do.

It is totally worth it. Being a pilot will give you a new outlook on life.

You’ll be amazed at what radios do a 3000ft. Have you ever had an 3000-foot antenna tower?

The world is glorious at 1000ft up.

Share your enthusiasm with those around you. You have a perspective very few ever see, except for a few seconds on the way to 35,000ft.

Earlier this month, I flew solo for the first time — the biggest milestone on the way to getting the pilot’s license. Here’s a photo my flight instructor took as I was coming in to land that day.

Today I took my first flight to another airport. It wasn’t far — about 20 miles away — but it was still a thrill. I flew about 1500ft above the ground, roughly above a freeway that happened to be my route. From that height, things still look three-dimensional. The grain elevator that marked out the one small town, the manufacturing plant at another, the college at the third. Bales of hay dotting the fields, the occasional tractor creeping along a road, churches sticking up above the trees. These are places I’ve known for decades, and now, suddenly, they are all new.

What a time to be alive! I am glad that our world is still so full of wonder and beauty.

]]>http://changelog.complete.org/archives/9364-true-things-about-learning-to-fly/feed29364First steps with smartcards under Linux and Android — hard, but it workshttp://changelog.complete.org/archives/9358-first-steps-with-smartcards-under-linux-and-android-hard-but-it-works
http://changelog.complete.org/archives/9358-first-steps-with-smartcards-under-linux-and-android-hard-but-it-works#commentsFri, 17 Jul 2015 03:04:23 +0000http://changelog.complete.org/?p=9358Continue reading First steps with smartcards under Linux and Android — hard, but it works→]]>Well this has been an interesting project.

It all started with a need to get better password storage at work. We wound up looking heavily at a GPG-based solution. This prompted the question: how can we make it even more secure?

Well, perhaps, smartcards. The theory is this: a smartcard holds your private keys in a highly-secure piece of hardware. The PC can never actually access the private keys. Signing and decrypting operations are done directly on the card to prevent the need to export the private key material to the PC. There are lots of “standards” to choose from (PKCS#11, PKCS#15, and OpenPGP card specs) that are relevant here. And there are ways to use SSH and OpenVPN with some of these keys too. Access to the card is protected by a passphrase (called a “PIN” in smartcard lingo, even though it need not be numeric). These smartcards might be USB sticks, or cards you pop into a reader. In any case, you can pop them out when not needed, pop them in to use them, and… well, pretty nice, eh?

So that’s the theory. Let’s talk a bit of reality.

First of all, it is hard for a person like me to evaluate how secure my data is in hardware. There was a high-profile bug in the OpenPGP JavaCard applet used by Yubico that caused the potential to use keys without a PIN, for instance. And how well protected is the key in the physical hardware? Granted, in most of these cards you’re talking serious hardware skill to compromise them, but still, this is unknown in absolute terms.

Here’s the bigger problem: compatibility. There are all sorts of card readers, but compatibility with pcsc-tools and pcscd on Linux seems pretty good. But the cards themselves — oh my. PKCS#11 defines an interface API, but each vendor would provide their own .so or .dll file to interface. Some cards (for instance, the ACOS5-64 mentioned on the Debian wiki!) are made by vendors that charge $50 for the privilege of getting the drivers needed to make them work… and they’re closed-source proprietary drivers at that.

The OpenPGP card and ACOS5-64 are the only ones in the list that support 4096-bit RSA keys due to the computational demands of them. The others all support 2048-bit RSA keys.

The JavaCOS requires the user to install a JavaCard applet to the card to make it useable. The Yubico OpenPGP applet works here, along with GlobalPlatform to install it. I am not sure just how solid it is. The Yubikey Neo has yet to arrive; it integrates some interesting OAUTH and TOTP capabilities as well.

I found that Debian’s wiki page for smartcards lists a bunch of them that are not really useable using the tools in main. The ACOS5-64 was such a dud. But I got the JavaCOS A22 working quite nicely. It’s also NFC-enabled and works perfectly with OpenKeyChain on Android (looking like a “Yubikey Neo” to it, once the OpenPGP applet is installed). I’m impressed! Here’s a way to be secure with my smartphone without revealing everything all the time.

Really the large amount of time is put into figuring out how all this stuff fits together. I’m getting there, but I’ve got a ways to go yet.

Update: Corrected to read “signing and decrypting” rather than “signing and encrypting” operations are being done on the card. Thanks to Benoît Allard for catching this error.

]]>http://changelog.complete.org/archives/9358-first-steps-with-smartcards-under-linux-and-android-hard-but-it-works/feed59358Roundup of remote encrypted deduplicated backups in Linuxhttp://changelog.complete.org/archives/9353-roundup-of-remote-encrypted-deduplicated-backups-in-linux
http://changelog.complete.org/archives/9353-roundup-of-remote-encrypted-deduplicated-backups-in-linux#commentsThu, 11 Jun 2015 17:09:51 +0000http://changelog.complete.org/?p=9353Continue reading Roundup of remote encrypted deduplicated backups in Linux→]]>Since I wrote last about Linux backup tools, back in a 2008 article about BackupPC and similar toools and a 2011 article about dedpulicating filesystems, I’ve revisited my personal backup strategy a bit.

I still use ZFS, with my tool “simplesnap” that I wrote about in 2014 to perform local backups to USB drives, which get rotated offsite periodically. This has the advantage of being very fast and very secure, but I also wanted offsite backups over the Internet. I began compiling criteria, which ran like this:

Remote end must not need any special software installed. Storage across rsync, sftp, S3, WebDAV, etc. should all be good candidates. The remote end should not need to support hard links or symlinks, etc.

Cross-host deduplication at at least the file level is required, so if I move a 4GB video file from one machine to another, my puny DSL wouldn’t have to re-upload it.

All data that is stored remotely must be 100% encrypted 100% of the time. I must not need to have any trust at all in the remote end.

Each backup after the first must send only an incremental’s worth of data across the line. No periodic re-uploading of the entire data set can be done.

The repository format must be well-documented and stable.

So, how did things stack up?

Didn’t meet criteria

A lot of popular tools didn’t meet the criteria. Here are some that I considered:

BackupPC requires software on the remote end and does not do encryption.

None of the rsync hardlink tree-based tools are suitable here.

rdiff-backup requires software on the remote end and does not do encryption or dedup.

duplicity requires a periodic re-upload of a full backup, or incremental chains become quite long and storage-inefficient. It also does not support dedup, although it does have an impressive list of “dumb” storage backends.

ZFS, if used to do backups the efficient way, would require software to be installed on the remote end. If simple “zfs send” images are used, the same limitations as with duplicity apply.

The tools must preserve POSIX attributes like uid/gid, permission bits, symbolic links, hard links, etc. Support for xattrs is also desireable but not required.

bup and zbackup are both interesting deduplicators, but do not yet have support for removing old data, so are impractical for this purpose.

burp requires software on the server side.

Obnam and Attic/Borg Backup

Obnam and Attic (and its fork Borg Backup) are both programs that have a similar concept at their heart, which is roughly this: the backup repository stores small chunks of data, indexed by a checksum. Directory trees are composed of files that are assembled out of lists of chunks, so if any given file matches another file already in the repository somewhere, the added cost is just a small amount of metadata.

Obnam was eventually my tool of choice. It has built-in support for sftp, but its reliance on local filesystem semantics is very conservative and it works fine atop davfs2 (and, I’d imagine, other S3-backed FUSE filesystems). Obnam’s repository format is carefully documented and it is very conservatively designed through and through — clearly optimized for integrity above all else, including speed. Just what a backup program should be. It has a lot of configurable options, including chunk size, caching information (dedup tables can be RAM-hungry), etc. These default to fairly conservative values, and the performance of Obnam can be significantly improved with a few simple config tweaks.

Attic was also a leading contender. It has a few advantages over Obnam, actually. One is that it uses an rsync-like rolling checksum method. This means that if you add 1 byte at the beginning of a 100MB file, Attic will upload a 1-byte chunk and then reference the other chunks after that, while Obnam will have to re-upload the entire file, since its chunks start at the beginning of the file in fixed sizes. (The only time Obnam has chunks smaller than its configured chunk size is with very small files or the last chunk in a file.) Another nice feature of Attic is its use of “packs”, where it groups chunks together into larger pack files. This can have significant performance advantages when backing up small files, especially over high-latency protocols and links.

On the downside, Attic has a hardcoded fairly small chunksize that gives it a heavy metadata load. It is not at all as configurable as Obnam, and unlike Obnam, there is nothing you can do about this. The biggest reason I avoided it though was that it uses a single monolithic index file that would have to be uploaded from scratch after each backup. I calculated that this would be many GB in size, if not even tens of GB, for my intended use, and this is just not practical over the Internet. Attic assumes that if you are going remote, you run Attic on the remote so that the rewrite of this file doesn’t have to send all the data across the network. Although it does work atop davfs2, this support seemed like an afterthought and is clearly not very practical.

Attic did perform much better than Obnam in some ways, largely thanks to its pack support, but the monolothic index file was going to make it simply impractical to use.

There is a new fork of Attic called Borg that may, in the future, address some of these issues.

Brief honorable mentions: bup, zbackup, syncany

There are a few other backup tools that people are talking about which do dedup. bup is frequently mentioned, but one big problem with it is that it has no way to delete old data! In other words, it is more of an archive than a backup tool. zbackup is a really neat idea — it dedups anything you feed at it, such as a tar stream or “zfs send” stream, and can encrypt, too. But it doesn’t (yet) support removing old data either.

syncany is fundamentally a syncing tool, but can also be used from the command line to do periodic syncs to a remote. It supports encryption, sftp, webdave, etc. natively, and runs on quite a number of platforms easily. However, it doesn’t store a number of POSIX attributes, such as hard links, uid/gid owner, ACL, xattr, etc. This makes it impractical for use for even backing up my home directory; I make fairly frequent use of ln, both with and without -s. If there were some tool to create/restore archives of metadata, that might work out better.

]]>http://changelog.complete.org/archives/9353-roundup-of-remote-encrypted-deduplicated-backups-in-linux/feed249353First impressions and review of OwnCloudhttp://changelog.complete.org/archives/9347-first-impressions-and-review-of-owncloud
http://changelog.complete.org/archives/9347-first-impressions-and-review-of-owncloud#commentsSat, 09 May 2015 01:57:54 +0000http://changelog.complete.org/?p=9347Continue reading First impressions and review of OwnCloud→]]>In my recent post (I give up on Google), a lot of people suggested using OwnCloud as a replacement for several Google services. I’ve been playing around with it for a few days, and it is something of a mix of awesome and disappointing, in my opinion.

Files

OwnCloud started as a file-sync tool, somewhat akin to Google Drive and Dropbox. It has clients for every platform, and it is also a client for every platform: you can have subfolders of your OwnCloud installation stored on WebDav, *FTP*, Google Drive, Dropbox, you name it. It is a pretty nice integrator of other storage services, and provides the only way to use some of them on Linux (*cough* Google Drive *cough*)

One particularly interesting feature is the live editing in the browser of ODT, DOCX, and TXT files. This is somewhat similar to Google Docs and the only such thing I’ve seen in Open Source software. It writes changes directly back to the documents and, in my limited testing, seems to work well. A very nice feature!

I’ve tested the syncing only on Linux so far, but it looks solid.

There are two surprising issues, however: there is no deduplication and no delta-uploads. Add 10 bytes to the end of a 1GB file, and you re-upload the 1GB file. Thankfully the OwnCloud GUI client is smart enough to use inotify to notice an mv, but my guess is — and I haven’t tested this, but apparently OwnCloud doesn’t use hashes at all — that the CLI client would require a reupload after any mv, because it doesn’t run continuously.

In some situations, Syncany may be a useful work-around for this, as it does chunk-based dedup and client-side encryption. However, you would lose a lot of the sharing features inside OwnCloud by doing this, and the integration with the OwnCloud “apps” for photos, videos, and music.

The Android/mobile apps support all the usual auto-upload options.

Calendar

A lot of people report using OwnCloud as a calendar server, and it does indeed use CalDAV. With a program like DAVDroid or Mozilla Lightning, this makes, in theory, a full-functioning calendar syncing tool. There is, of course, also a web interface to the calendar. It, sadly, is limited. Or shall we say, VERY limited. Even something like sending an invite is missing — and in fact, the GUI for sharing an event is baffling. You can share it with someone, they get no say in whether or not it shows up, and it shows up on their calendar on the web only (not on synced copies) and they have no way to remove it!

I haven’t tested this yet, but there’s not much to test, I suspect. It can be shared with others, which I could see as a nice feature.

Bookmarks

An interesting bookmarks manager, though mysteriously not with Firefox sync support. There is Chrome sync support, and a separate Mozilla Sync support, but it doesn’t provide cross-browser syncing, apparently.

Music

It is designed to present an interface to music that is stored in Files. It provides an Ampache-compatible API, so there are a lot of clients that can stream music. It has very few options, not even for transcoding, so I don’t see how it would be useful for my FLAC collection.

Pictures

Sort of a gallery view of photos synced up with Files. Very basic. Has a sharing button to share a link to an entire folder, but no option to embed photos in blog posts at a lower resolution or shortcut to sharing individual photos.

Notes, Tasks, etc.

I haven’t had the chance to look at this much. Some of them sync to various clients. The Notes are saved as HTML files that get synced down.

Clients overall

There is a very helpful page that lists all the sync clients for OwnCloud — not just for files, but also for calendars, contacts, etc. The list is extensive!

Other options

The two other Open Source options mentioned on my blog post were Kolab and Sogo, and there is also Zimbra which also has a community edition. The Debian Groupware page lists a number of other groupware options as well. Citadel caught my eye (wow, it’s still around!). Sogo has ActiveSync support, which might make phone integration a lot easier. It is not dead-simple to set up like OwnCloud is, though, so I haven’t tried it out, but I will probably be looking at it and Citadel next.

The most recent example being retiring Classic Maps. That’s a problem, because the current Maps mysteriously doesn’t show most of my saved (“starred”) places. Google has known about this since at least 2013. There are posts all over their forums about it going back to when what is now “regular” Google Maps was beta. Google employees even knew about it and did nothing. For someone that made heavy use of it, this was quite annoying.

Removed the 7-day and month views from Calendar for Android, claiming this was “better” for users. Finally re-added those views a few months later after many complaints. I even participated in a survey process with them where they were clearly struggling to understand why anybody wanted to see 7 days at once, when that feature had been there for years…

Removing the XMPP capabilities in Google Talk/Hangouts.

Picasaweb pretty much shut down, with very strong redirects to Google+ Photos. Which still to this day doesn’t have a handy feature for embedding in a blog post or anything that’s not, well, Google+.

General creeping crapification of everything they touch. It’s almost like Microsoft in the 90s all over again. All of a sudden my starred places stop showing up in Google Maps, but show up in Google Drive — shared with the whole world. What? I never wanted them in Google Drive to start with.

All the products that are all-but-dead — Google Groups and the sad state of the Deja News archives. Maybe Google+ itself goes on this list soon?

Looks like they’re trying to kill off Google Voice and merge it into hangouts, but I can’t send a text from the web with Hangouts.

And this massive list of discontinued services and products. Yeowch. Remember when Google Code was hot, and then they didn’t touch it at all for years?

And they still haven’t fixed some really basic things, such as letting people change their email address when they get married.

Dropping SIP from Grand Central, ActiveSync from Apps, etc.

I even used to use Flickr, then moved to Picasa when Yahoo stopped investing in Flickr. Now I’m back to Flickr, because Google stopped investing in Picasa.

The takeaway is that you can’t really rely on Google for anything. Counting on something being there for an upcoming trip and then having it be suddenly yanked away is a level of frustration that just makes the service not so useful. Never knowing when obvious things (7-day calendar view) will be removed means you just can’t depend on it.

So, are there good alternatives? Things I’m thinking of include:

Alternative calendar applications. Ideally it would support shared calendars for multiple people in a family, an Android app that lets you easily view some or all calendars, etc. I wonder if outlook.com is really the only competitor here? Last I looked — a few years ago — none of the Open Source options really worked well.

Alternative mapping applications. Must-haves include directions, navigation in the car, saving points of interest, and offline storage on Android. Nice-to-haves would include restaurant review integration, etc. Looks like Nokia (HERE.com) and Mapquest, plus a few OSM spinoffs, are the leading contenders here.

Email is easily enough found elsewhere, and I’ve never used Gmail much anyhow.

“For once you have tasted flight,
You will walk the earth with your eyes turned skyward;
For there you have been,
And there you long to return.”

– Leonardo da Vinci

There is something of a magic to flight, to piloting. I remember the first flight I ever took, after years of dreaming of flying in a plane: my grandma had bought me a plane ticket. In one of the early morning flights, I witnessed a sunrise above cumulus clouds. Although I was just 10 or so at the time, that still is a most beautiful image seared into my memory.

I have become “meh” about commercial flight over the years. The drive to the airport, the security lines, the lack of scenery at 35,000 feet. And yet, there is much more to flight than that. When I purchased what was essentially a flying camera, I saw a whole new dimension of the earth’s amazing beauty. All the photos in this post, in fact, are ones I took. I then got a RC airplane, because flying the quadcopter was really way too easy.

“It’s wonderful to climb the liquid mountains of the sky.
Behind me and before me is God, and I have no fears.”

– Helen Keller

Start talking to pilots, and you notice a remarkable thing: this group of people that tends to be cool and logical, methodical and precise, suddenly finds themselves using language almost spiritual. Many have told me that being a pilot brings home how much all humans have in common, the unifying fact of sharing this beautiful planet together. Many volunteer with organizations such as Angel Flight. And having been up in small planes a few times, I start to glimpse this. Flying over my home at 1000′ up, or from lake to lake in Seattle with a better view than the Space Needle, seeing places familiar and new, but from a new perspective, drives home again and again the beauty of our world, the sheer goodness of it, and the wonderful color of the humanity that inhabits it.

“The air up there in the clouds is very pure and fine, bracing and delicious.

And why shouldn’t it be?

It is the same the angels breathe.”

– Mark Twain

The view from 1000 feet, or 3000, is often so much more spectacular than the view from 35,000 ft as you get on a commercial flight. The flexibility is too; there are airports all over the country that smaller planes can use which the airlines never touch.

Here is one incredible video from a guy that is slightly crazy but does ground-skimming, flying just a few feet off the ground: (try skipping to 9:36)

So what comes next is something I blame slightly on my dad and younger brother. My dad helped get me interested in photography as a child, and that interest has stuck. It’s what caused me to get into quadcopters (“a flying camera for less than the price of a nice lens!”). And my younger brother started mentioning airplanes to me last year for some reason, as if he was just trying to get me interested. Eventually, it worked. I started talking to the pilots I know (I know quite a few; there seems to be a substantial overlap between amateur radio and pilots). I started researching planes, flight, and especially safety — the most important factor.

And eventually I decided I wanted to be a pilot. I’ve been studying feverishly, carrying around textbooks and notebooks in the car, around the house, and even on a plane. There is a lot to learn.

And today, I took my first flight with a flight instructor. Today I actually flew a plane for awhile. Wow! There is nothing quite like that experience. Seeing a part of the world I am familiar with from a new perspective, and then actually controlling this amazing machine — I really fail to find the words to describe it. I have put in many hours of study already, and there will be many more studying and flying, but it is absolutely worth it.

Here is one final video about one of the most unique places you can fly to in Kansas.

]]>http://changelog.complete.org/archives/9337-today-i-flew-a-plane/feed09337ssh suddenly stops communicating with some hostshttp://changelog.complete.org/archives/9335-ssh-suddenly-stops-communicating-with-some-hosts
http://changelog.complete.org/archives/9335-ssh-suddenly-stops-communicating-with-some-hosts#commentsMon, 30 Mar 2015 22:13:21 +0000http://changelog.complete.org/?p=9335Continue reading ssh suddenly stops communicating with some hosts→]]>Here’s a puzzle I’m having trouble figuring out. This afternoon, ssh from my workstation or laptop stopped working to any of my servers (at OVH). The servers are all running wheezy, the local machines jessie. This happens on both my DSL and when tethered to my mobile phone. They had not applied any updates since the last time ssh worked. When looking at it with ssh -v, they were all hanging after:

Now, I tried rebooting machines at multiple ends of this. No change. I tried connecting from multiple networks. No change. And then, as I was writing this blog post, all of a sudden it works normally again. Supremely weird! Any ideas what I can blame here?