Avtok

Much has been written about Go’s interfaces. They’re one of
the language features I like best, providing a very pragmatic trade-off between
the complexity of more powerful type systems and the anarchy of duck typing. If
you’re not already familiar with Go or Go’s interfaces, I’d encourage you to
read an introduction to Go and to play around with the language a
bit—it’ll help you better understand the remainder of this post.

Interfaces in Go are most commonly used as a means of encapsulation. They allow
programmers to hide an object’s underlying state, only exposing a carefully
curated API. They also allow objects to be decoupled in a way that allows them
to be easily replaced by alternate implementations or stubbed out entirely.

However, Go’s interfaces—and in particular, interface conversions—allow for
a more interesting and less obvious pattern that I call “interface upgrades.”
The crucial observation here is that not only can interfaces be safely cast to
narrower interfaces (i.e., every io.ReadCloser is also an
io.Reader), but they can also be cast to wider or even unrelated
interfaces if their dynamic types support it.1

The reason I call these sorts of casts interface upgrades is that they remind me
of protocol upgrades, e.g., the one performed during HTTP to negotiate the use
of web sockets. In both cases, two cooperating parties communicating over one
specified protocol (HTTP in one case, and some interface type in the other) can
attempt to switch to another protocol with a different set of features.

A good place to search for Go examples is its standard library, so in order to
illustrate interface upgrades, let’s dive into three examples taken from the
source of Go itself.

Efficient io

Like all articles about Go’s interfaces, we are obligated to start with Go’s
io package.

The io package is essentially a set of protocols for copying bytes around.
Unfortunately, this is quite slow, and therefore it’s in Go’s best interest to
try to move each byte as few times as possible.

There’s quite possibly no better place to attack this problem than
io.Copy, Go’s powerhouse of byte moving. This function takes an
io.Reader and an io.Writer, and moves data from one to
the other. Simple enough.

But there’s a catch: both io.Reader’s Read and io.Writer’s Write take
buffers as arguments: they expect their caller to provide the necessary memory
for them. The only sensible implementation of io.Copy, then, is to allocate a
buffer and pass it alternately to Read and Write until all the data has been
copied. The end result is that every byte gets copied twice: once when Read
places it into the Copy-internal buffer, and once when Write removes it.

In many cases we can do better, however. If we’re writing to a file from an
internal bytes.Buffer, for instance, there’s no need to allocate an
intermediary buffer at all: we can pass bytes.Buffer’s internal buffer to
syscall.Write directly, turning a two-copy process into a single-copy
one.

Supporting this kind of copy elision in io.Copy is tricky since we can’t
change the signature of either io.Reader.Read or io.Writer.Write without
sacrificing generality. Luckily, interface upgrades come to our rescue. The io
package defines two auxiliary types, io.WriterTo and
io.ReaderFrom which io.Readers and io.Writers (respectively)
may optionally implement. If at least one of the sides of any given io.Copy
can be upgraded to one of these alternate interfaces, they can arrange to copy
data directly between their respective buffers (or to use the other’s buffers
directly), eliminating the need for io.Copy to allocate buffers of its own.

You’ll find that many of Go’s built-in buffer types (e.g., those in package
bufio, as well as bytes.Buffer and strings.Reader)
also implement io.ReaderFrom and io.WriterTo, allowing many common io
pipelines to be performed with less byte movement than one might expect.

Extending net/http

Interface upgrades can also be used to add functionality to existing interfaces,
especially when you are unable (or unwilling) to modify an existing
interface.

A good example of this is net/http’sResponseWriter. In particular, net/http exposes three
additional interfaces, CloseNotifier, Flusher,
and Hijacker, each of which augment the capabilities of a vanilla
ResponseWriter. The default implementation of a ResponseWriter incidentally
supports all three of these additional interfaces (although nothing in the
exposed types would tell you that), and using type assertions to upgrade a
http.ResponseWriter to one of these other interfaces allows you to unlock this
additional functionality.

For instance, if you wanted to flush an HTTP response body halfway through, you
might write:

The variable w is a http.ResponseWriter and does not ordinarily support the
Flush() function, but by performing a type assertion to
http.Flusher you can upgrade to a interface type that does.

For library authors, this sort of interface upgrade is pretty neat, since it
allows them to provide additional functionality in a backwards-compatible way.
By simply defining additional methods on some interface’s dynamic type (in the
case of net/http.ResponseWriter, this is a private struct named
response) you can let people opt in to new methods without
breaking any existing code.

For consumers of libraries, including standard ones like net/http, the story
isn’t quite as great. While it just so happens that all the ResponseWriters
that net/http give you implement (e.g.) CloseNotifier, there’s not really
any way for you to know that without reading the source. The type system
certainly can’t tell you this (almost by design), and even the
documentation—which is normally very good—falls short here. This is
by no means inherent to interface upgrades themselves (compare
CloseNotifier’s documentation to, for instance,
io.Copy’s), but without a concious effort to document supported
upgrades they’re likely to remain largely unused.

This is also probably a good place to mention that interface upgrades put you
beyond the safety of the type checker. If you write an invalid upgrade either
due to programmer error or due to a change in a library you’re using, the way
you’ll find out is when your program panics at runtime. Therefore, you should
always use the “comma, ok” idiom when performing type assertions, and always
provide fallback behavior for when you discover that an upgrade cannot be
made.2

Optimizing net

As I mentioned above, moving data around is pretty slow, and a good way to make
programs faster is to avoid copies at all. Unfortunately, the normal read and
write cycle forces us to do at least two copies (regardless of whatever
io.Copy tricks we do in Go): once from kernel space to receive data, and once
to kernel space to send it somewhere else.

Just like before, we can sometimes do better. Certain operating systems expose
system calls like sendfile(2) (available in some form on most
UNIX systems; a similar TransmitFile mechanism exists on Windows), which
instructs the kernel to move data to and from certain sorts of file descriptors
with a single kernel-internal copy operation, an improvement over the ordinary
two kernel-to-userspace copies.

Making use of these more efficient system calls often requires careful
application-level bookkeeping and invasive architectural changes, and as a
result only high-performance HTTP servers typically use them. Go, however, can
simply leverage its existing abstractions, treating this as an interface upgrade
like any other.3

Let’s dive in to the source of net to have a look. net.TCPConn,
the type underlying every TCP connection, defines a function which should now be
familiar to you: io.ReaderFrom’s ReadFrom. You’ll notice that
this function first attempts to call a function named sendFile, falling back
to a generic ReadFrom implementation if that didn’t work.

sendFile is where all the interesting work happens. Since the
sendfile(2) system call only works when sending regular files (and not, for
instance, a bytes.Buffer or any other io.Reader), sendFile first attempts
two type assertions to determine if the io.Reader it was passed happens to be
a os.File, possibly wrapped in an io.LimitedReader. If
so, the io.Reader is (potentially) eligible, and sendfile(2) is used to
shuttle bytes around.

Let’s stop here for a second and think about how cool this is. The Go standard
library has automatically, and in all likelihood without you knowing, upgraded
every io.Copy from a file to a TCP socket to use a system call that was
previously reserved for high-performance proxies, just because it could.

But that’s not even the most amazing thing. By making this upgrade path rely on
the standard io.ReaderFrom interface, Go has given us enough rope to allow
anyio.Writer that wraps a net.TCPConn to take advantage of this
optimization, just by enabling an interface upgrade to io.ReaderFrom.

So let’s tie this entire post together by going back and having another look at
http.ResponseWriter. We previously looked at how it offered additional
functionality through interface upgrades. What we overlooked at the time was the
fact that the default http.ResponseWriter actually implements one more method:
ReadFrom.

So not only does net perform an interface upgrade to support sendfile(2)
when it can, but net/http is able to support the same upgrade to make every
io.Copy to an http.ResponseWriter—for instance, the one in the standard
http.ServeFile function—also support sendfile(2). Because
of an unlikely combination of well-designed interface types and the ability to
upgrade to more efficient interfaces when necessary, Go is able to serve files
as efficiently as nginx without your knowledge or cooperation.

And that’s fucking amazing.

The Proxy Problem

Let’s again look at http.ResponseWriter. Let’s say we want to write a request
logger middleware that captures and prints out the HTTP status code that we
returned. A naive solution might look like this:

Let’s set aside all the ways in which the code above is subtly incorrect and
focus on the interaction between the StatusLogger and the
http.ResponseWriter it wraps. Our intent is fairly clear: we want to intercept
all calls to WriteHeader, saving the status code that is sent, but otherwise
act exactly as the object we’re proxying.

Unfortunately, our StatusLogger only implements the bare minimum required of
an http.ResponseWriter, and in particular doesn’t implement any interface
upgrades. Users who are expecting upgradable functionality (including those that
rely on the performance characteristics of, e.g., sendfile(2) upgrades) will
find their applications broken in subtle and in all likelihood, hard-to-debug
ways.

In general, it’s impossible to know which interface upgrades a given object
supports, since doing so would require deep knowledge of the method set of a
interface’s dynamic type. It’s therefore also impossible to write a
general-purpose proxy type that supports arbitrary interface upgrades. Even if
we have only a small number of interface upgrades to support (for example.
http.ResponseWriter’s four), supporting all combinations of those interfaces
becomes an unwieldy power set (http.ResponseWriter would require sixteen
implementations).

But in practice it’s possible to do pretty well, especially for standard library
types like http.ResponseWriters. Here, for any given version of Go we have a
single well-known set of upgrades to support. In this case, there are two common
configurations: a bare http.ResponseWriter, and one with all the fancy bells
and whistles. If you want to see this technique at work, I’ve written a generic,
interface upgrade aware http.ResponseWriter proxy as part of
Goji.

Use Sparingly

Interface upgrades allow for some of the coolest behaviors in Go, but I’m going
to end this post on a somewhat unusual note: I’d strongly encourage you not to
write your own.

Interface upgrades derive a lot of their utility from being standard. Just like
a network protocol upgrade, both sides need to agree on what the resulting
behavior is. There are a few well-known interface upgrades in Go’s standard
library which I will happily promote the use of (particularly the ones in io),
but chances are your library isn’t standard enough to make good use of a custom
interface upgrade. The downsides of sidestepping the type system are very real,
and I doubt the complexity will often be worth it.

It took me a while to truly appreciate Go’s interfaces: they manage to hide a
remarkable amount of complexity for such a simple idea. It’s really amazing how
much power you can get from just a few well thought out interfaces and the means
to convert between them, and it’s a constant reminder of how well designed Go’s
standard library is.

I’m sure I’m not the first to notice this phenomenon, which means that
someone else has probably already given it a different name. If you know
of any interesting articles about this from the literature or elsewhere on
the internet, please send them my way. ↩

If for whatever reason this is impossible in your application (for
instance, you rely on being able to Hijack() a http.ResponseWriter),
you should be sure to have “pre-flight” runtime checks to ensure all the
interface upgrades you rely on are supported. ↩

There’s actually a second system call available on Linux,
splice(2), which allows the efficient transfer between an
arbitrary file descriptor and a pipe. By splicing one file descriptor to
one end of a pipe and another file descriptor to the other end, you can
use this system call to perform arbitrary fd-to-fd copies without ever
involving userspace. Unlike sendfile(2), Go doesn’t currently support
this using interface upgrades, but I can’t think of a reason it couldn’t. ↩

HFS+, the filesystem used on modern versions of Mac OS, is long past its prime.
It’s actually a remarkable feat of engineering: the same filesystem has been
extended and improved upon for over ten years despite technical limitations
inherent in its original design. However, sometimes its limitations are a little
too much.

I have kept a checkout of the linux 2.6 git repository on my Mac for some time
now – it’s sometimes fun to go sourcediving through the kernel, even though I
grok very little of it. A few months ago, however, I noticed that after a pull,
git said I had unstaged changes in my working directory. This was curious
since I hadn’t changed anything, but I didn’t think much of it, and after a few
failed git reset --hards, I forgot about it.

Today, however, I got frustrated at git and decided to dig further into the
problem. I tried everything – first I tried adding and committing the unstaged
changes, but git add refused to add any of the supposedly untracked files.
git reset --hard and git clean were similarly ineffective. git rm -f was
the only command that I could actually get to function properly, but that put
git in some odd state in which the files were both staged and not staged for
deletion.

But then I looked closer at the filenames. Turns out they weren’t quite the
same. The set of files that was staged for deletion were partially uppercase,
while the unstaged ones were entirely lowercase:

Suddenly it clicked. HFS+, by default, is a case-insensitve filesystem. At
some point in the last few months, the Linux repo added multiple files with
different capitalizations, pretty much screwing over anyone still stuck with
this miserable excuse for a filesystem.

GNU screen is a fantastic little program that allows you to multiplex many
terminals (think tabs) in a single terminal, especially an ssh session. It
also allows you to detatch from and reconnect to this session later, even after
your ssh connection has died.

Learning screen isn’t hard (and there are plenty of
resources on the internet to help you). Unfortunately, the most
natural way to use screen is surprisingly mediocre, especially if you use
ssh agent forwarding.

Starting a screen

Starting a screen session isn’t rocket science: just ssh to your server of
choice and type screen. However, this isn’t a great way of doing things.
First, you have to remember whether you already have a screen open on this
server (if so, you need to use screen -d -r instead). Second, by ssh-ing,
you’ve stared a shell that you will never use (in fact, should you ever find
yourself using this shell, you’re probably going to be very sad when ssh
inevitably flakes out on you).

We can fix both of these problems pretty easily. First, a quick peek at
screens man page shows some interesting flags:

-d -R — Attach here and now. In detail this means: If a session is running,
then reattach. If necessary detach and logout remotely first. If it was not
running create it and notify the user. This is the author’s favorite.

-d -RR — Attach here and now. Whatever that means, just do it. (This one
seems to do the same thing but sounds more awesome, so it’s my favorite)

-x — Attach to a not detached screen session. (Multi display mode).

-U — Run screen in UTF-8 mode. This option tells screen that your
terminal sends and understands UTF-8 encoded characters. It also sets the
default encoding for new windows to ‘utf8’.

We arrive at our final screen invocation, screen -UxRR. This will reattach
to a screen should one already exist (but not detatching the other session),
or create a new session otherwise. We also force everything into UTF-8 mode,
because there’s really no reason not to.

As for the pesky superfluous shell we’ve spawned, we can turn to a few useful
ssh options. ssh example.com [commandname] will run a specified command on
the remote machine instead of your default shell, but it’ll complain that it
“Must be connected to a terminal.” The solution here is to force ssh to
allocate a TTY for the session with the -t switch.

We’re finally left with a reasonable way to ssh to a remote computer:

ssh -t example.com screen -UxRR

But what about my Agent?

ssh-agent is another really neat tool that will allow you to authenticate
against your private key without typing in your key’s passphrase every time.
It’s a great way to regain the ease-of-use of a passwordless key while actually
having Security and other Nice Things (tm).

On its own, ssh-agent is a fantastic tool. However, combined with a feature
called agent forwarding, it becomes a true powerhouse. The idea is that you
allow certain hosts you’re sshed into to forward their ssh-agent requests to
the agent running on your home computer. In effect, this allows you to do
passwordless ssh-jumping (if, for instance, you have to connect to a gateway
machine to reach an internal protected network, or if you need to supply
credentials to a git server in order to commit).

Note: You should be aware of the security model of agent forwarding. In
particular, anyone with root access to any machine you forward your agent to
will, for the duration of your connection, be able to authenticate to other
machines as you. Therefore, make sure you only forward your agent to machines
that you completely trust.

However, screen is notorious for breaking agent forwarding. If a screen was
created with one set of agent sockets, but that ssh connection has long since
died, where should ssh-agent requests go?

Thankfully, the Internet has come up with severalsolutions
to this problem. Unfortunately, none of them are very good—they’re generally
downright hideous hacks, and in particular none of them seem to work with our
improved screen invocation above (they rely on having a login shell).

Finding a non-hacky solution to this problem requires a bit of doing. First,
let’s look at how ssh-agent works. If you try spawning a new ssh-agent,
it’ll spit out something like this (linebreaks added for clarity)

When we forward our agent, essentially all that’s happening is that a socket is
set up in some random location in /tmp on the remote machine that shuttles
bytes back and forth between the remote machine and the UNIX socket in /tmp on
our local computer. The issue for screen, then, is that this random path
changes every time we connect.

Instead of keeping each screen window aware of the current socket in /tmp, as
most solutions suggest, it’s easier to keep the socket in a single known
location, like ~/.authsock. First, let’s tell screen about this new socket.
Add the following to your .screenrc:

setenv SSH_AUTH_SOCK "$HOME/.authsock"

Now we need to ensure we keep this socket up-to-date every time we connect. For
this, we symlink this path to the real socket on every connection. Normally, you
might do this in your bash profile, but since we are bypassing bash entirely
when we log in, that won’t work. Instead, we add the following to our
~/.ssh/rc (a file that ssh helpfully runs after every connection):

test $SSH_AUTH_SOCK && \
ln -sf "$SSH_AUTH_SOCK" "$HOME/.authsock"

This just symlinks the appropriate socket to ~/.authsock should you be
connecting with agent forwarding.

Mystery Hunt is an annual MIT tradition. Every year, about a thousand people
split over some 40-or-so teams descend on MIT to solve some wickedly hard
puzzles over the course of a weekend. And by weekend, I mean the whole weekend.
None of that “sleep” stuff.

I got to Team Codex headquarters on Friday morning. The main room reminded me of
Alfred Hitchcock’s Birds – literally every surface was covered with someone’s
computer. I eventually found enough space for me to squat awkwardly in a chair
with my laptop before we were off.

I started out on a puzzle that had me identifying bridges over the Charles
River. It was plenty of fun, and between Google Street View (super-useful, even
though it can be kind of sketchy too) and my somewhat patchy knowledge of
Cambridge, I managed to identify a bunch of them.

Over the coming hours, I worked on trying to brute-force a particularly nasty
constraints problem using Ruby (the humans beat me to it), attempting a puzzle
involving frisbees and numbers that nobody ever figured out, and finding a word
that corresponded to the clue “gambling top” (“teetotum,” in case you’re
wondering), as well as miscellaneous legwork for other puzzles.

All in all, it was a great weekend, and I can’t wait until next year’s hunt.