Avtok

Interface Upgrades in Go

November 5, 2014

Much has been written about Go’s interfaces. They’re one of the language features I like best, providing a very pragmatic trade-off between the complexity of more powerful type systems and the anarchy of duck typing. If you’re not already familiar with Go or Go’s interfaces, I’d encourage you to read an introduction to Go and to play around with the language a bit—it’ll help you better understand the remainder of this post.

Interfaces in Go are most commonly used as a means of encapsulation. They allow programmers to hide an object’s underlying state, only exposing a carefully curated API. They also allow objects to be decoupled in a way that allows them to be easily replaced by alternate implementations or stubbed out entirely.

However, Go’s interfaces—and in particular, interface conversions—allow for a more interesting and less obvious pattern that I call “interface upgrades.” The crucial observation here is that not only can interfaces be safely cast to narrower interfaces (i.e., every io.ReadCloser is also an io.Reader), but they can also be cast to wider or even unrelated interfaces if their dynamic types support it.1

The reason I call these sorts of casts interface upgrades is that they remind me of protocol upgrades, e.g., the one performed during HTTP to negotiate the use of web sockets. In both cases, two cooperating parties communicating over one specified protocol (HTTP in one case, and some interface type in the other) can attempt to switch to another protocol with a different set of features.

A good place to search for Go examples is its standard library, so in order to illustrate interface upgrades, let’s dive into three examples taken from the source of Go itself.

Efficient io

Like all articles about Go’s interfaces, we are obligated to start with Go’s io package.

The io package is essentially a set of protocols for copying bytes around. Unfortunately, this is quite slow, and therefore it’s in Go’s best interest to try to move each byte as few times as possible.

There’s quite possibly no better place to attack this problem than io.Copy, Go’s powerhouse of byte moving. This function takes an io.Reader and an io.Writer, and moves data from one to the other. Simple enough.

But there’s a catch: both io.Reader’s Read and io.Writer’s Write take buffers as arguments: they expect their caller to provide the necessary memory for them. The only sensible implementation of io.Copy, then, is to allocate a buffer and pass it alternately to Read and Write until all the data has been copied. The end result is that every byte gets copied twice: once when Read places it into the Copy-internal buffer, and once when Write removes it.

In many cases we can do better, however. If we’re writing to a file from an internal bytes.Buffer, for instance, there’s no need to allocate an intermediary buffer at all: we can pass bytes.Buffer’s internal buffer to syscall.Write directly, turning a two-copy process into a single-copy one.

Supporting this kind of copy elision in io.Copy is tricky since we can’t change the signature of either io.Reader.Read or io.Writer.Write without sacrificing generality. Luckily, interface upgrades come to our rescue. The io package defines two auxiliary types, io.WriterTo and io.ReaderFrom which io.Readers and io.Writers (respectively) may optionally implement. If at least one of the sides of any given io.Copy can be upgraded to one of these alternate interfaces, they can arrange to copy data directly between their respective buffers (or to use the other’s buffers directly), eliminating the need for io.Copy to allocate buffers of its own.

You’ll find that many of Go’s built-in buffer types (e.g., those in package bufio, as well as bytes.Buffer and strings.Reader) also implement io.ReaderFrom and io.WriterTo, allowing many common io pipelines to be performed with less byte movement than one might expect.

Extending net/http

Interface upgrades can also be used to add functionality to existing interfaces, especially when you are unable (or unwilling) to modify an existing interface.

A good example of this is net/http’s ResponseWriter. In particular, net/http exposes three additional interfaces, CloseNotifier, Flusher, and Hijacker, each of which augment the capabilities of a vanilla ResponseWriter. The default implementation of a ResponseWriter incidentally supports all three of these additional interfaces (although nothing in the exposed types would tell you that), and using type assertions to upgrade a http.ResponseWriter to one of these other interfaces allows you to unlock this additional functionality.

For instance, if you wanted to flush an HTTP response body halfway through, you might write:

func(w http.ResponseWriter, r *http.Request) {
        io.WriteString(w, "It's going to be legen... ")
        io.WriteString(w, "(wait for it) ")
        if fl, ok := w.(http.Flusher); ok {
                fl.Flush()
                time.Sleep(1 * time.Second)
        }
        io.WriteString(w, "...dary. Legendary!\n")
}

The variable w is a http.ResponseWriter and does not ordinarily support the Flush() function, but by performing a type assertion to http.Flusher you can upgrade to a interface type that does.

For library authors, this sort of interface upgrade is pretty neat, since it allows them to provide additional functionality in a backwards-compatible way. By simply defining additional methods on some interface’s dynamic type (in the case of net/http.ResponseWriter, this is a private struct named response) you can let people opt in to new methods without breaking any existing code.

For consumers of libraries, including standard ones like net/http, the story isn’t quite as great. While it just so happens that all the ResponseWriters that net/http give you implement (e.g.) CloseNotifier, there’s not really any way for you to know that without reading the source. The type system certainly can’t tell you this (almost by design), and even the documentation—which is normally very good—falls short here. This is by no means inherent to interface upgrades themselves (compare CloseNotifier’s documentation to, for instance, io.Copy’s), but without a concious effort to document supported upgrades they’re likely to remain largely unused.

This is also probably a good place to mention that interface upgrades put you beyond the safety of the type checker. If you write an invalid upgrade either due to programmer error or due to a change in a library you’re using, the way you’ll find out is when your program panics at runtime. Therefore, you should always use the “comma, ok” idiom when performing type assertions, and always provide fallback behavior for when you discover that an upgrade cannot be made.2

Optimizing net

A final, rather astounding example can be found in Go’s net package.

As I mentioned above, moving data around is pretty slow, and a good way to make programs faster is to avoid copies at all. Unfortunately, the normal read and write cycle forces us to do at least two copies (regardless of whatever io.Copy tricks we do in Go): once from kernel space to receive data, and once to kernel space to send it somewhere else.

Just like before, we can sometimes do better. Certain operating systems expose system calls like sendfile(2) (available in some form on most UNIX systems; a similar TransmitFile mechanism exists on Windows), which instructs the kernel to move data to and from certain sorts of file descriptors with a single kernel-internal copy operation, an improvement over the ordinary two kernel-to-userspace copies.

Making use of these more efficient system calls often requires careful application-level bookkeeping and invasive architectural changes, and as a result only high-performance HTTP servers typically use them. Go, however, can simply leverage its existing abstractions, treating this as an interface upgrade like any other.3

Let’s dive in to the source of net to have a look. net.TCPConn, the type underlying every TCP connection, defines a function which should now be familiar to you: io.ReaderFrom’s ReadFrom. You’ll notice that this function first attempts to call a function named sendFile, falling back to a generic ReadFrom implementation if that didn’t work.

sendFile is where all the interesting work happens. Since the sendfile(2) system call only works when sending regular files (and not, for instance, a bytes.Buffer or any other io.Reader), sendFile first attempts two type assertions to determine if the io.Reader it was passed happens to be a os.File, possibly wrapped in an io.LimitedReader. If so, the io.Reader is (potentially) eligible, and sendfile(2) is used to shuttle bytes around.

Let’s stop here for a second and think about how cool this is. The Go standard library has automatically, and in all likelihood without you knowing, upgraded every io.Copy from a file to a TCP socket to use a system call that was previously reserved for high-performance proxies, just because it could.

But that’s not even the most amazing thing. By making this upgrade path rely on the standard io.ReaderFrom interface, Go has given us enough rope to allow any io.Writer that wraps a net.TCPConn to take advantage of this optimization, just by enabling an interface upgrade to io.ReaderFrom.

So let’s tie this entire post together by going back and having another look at http.ResponseWriter. We previously looked at how it offered additional functionality through interface upgrades. What we overlooked at the time was the fact that the default http.ResponseWriter actually implements one more method: ReadFrom.

So not only does net perform an interface upgrade to support sendfile(2) when it can, but net/http is able to support the same upgrade to make every io.Copy to an http.ResponseWriter—for instance, the one in the standard http.ServeFile function—also support sendfile(2). Because of an unlikely combination of well-designed interface types and the ability to upgrade to more efficient interfaces when necessary, Go is able to serve files as efficiently as nginx without your knowledge or cooperation.

And that’s fucking amazing.

The Proxy Problem

Interface upgrades aren’t without their faults. Besides the aforementioned problems for library users, interface upgrades present an enormous burden on authors of proxy objects.

Let’s again look at http.ResponseWriter. Let’s say we want to write a request logger middleware that captures and prints out the HTTP status code that we returned. A naive solution might look like this:

type StatusLogger struct {
        http.ResponseWriter
        StatusCode int
}

func (s *StatusLogger) WriteHeader(code int) {
        s.StatusCode = code
        s.ResponseWriter.WriteHeader(code)
}

func RequestLogger(h http.Handler) http.Handler {
        fn := func(w http.ResponseWriter, r *http.Request) {
                sl := &StatusLogger{ResponseWriter: w}
                h.ServeHTTP(sl, r)
                log.Println("HTTP status code", sl.StatusCode)
        }
        return http.HandlerFunc(fn)
}

Let’s set aside all the ways in which the code above is subtly incorrect and focus on the interaction between the StatusLogger and the http.ResponseWriter it wraps. Our intent is fairly clear: we want to intercept all calls to WriteHeader, saving the status code that is sent, but otherwise act exactly as the object we’re proxying.

Unfortunately, our StatusLogger only implements the bare minimum required of an http.ResponseWriter, and in particular doesn’t implement any interface upgrades. Users who are expecting upgradable functionality (including those that rely on the performance characteristics of, e.g., sendfile(2) upgrades) will find their applications broken in subtle and in all likelihood, hard-to-debug ways.

In general, it’s impossible to know which interface upgrades a given object supports, since doing so would require deep knowledge of the method set of a interface’s dynamic type. It’s therefore also impossible to write a general-purpose proxy type that supports arbitrary interface upgrades. Even if we have only a small number of interface upgrades to support (for example. http.ResponseWriter’s four), supporting all combinations of those interfaces becomes an unwieldy power set (http.ResponseWriter would require sixteen implementations).

But in practice it’s possible to do pretty well, especially for standard library types like http.ResponseWriters. Here, for any given version of Go we have a single well-known set of upgrades to support. In this case, there are two common configurations: a bare http.ResponseWriter, and one with all the fancy bells and whistles. If you want to see this technique at work, I’ve written a generic, interface upgrade aware http.ResponseWriter proxy as part of Goji.

Use Sparingly

Interface upgrades allow for some of the coolest behaviors in Go, but I’m going to end this post on a somewhat unusual note: I’d strongly encourage you not to write your own.

Interface upgrades derive a lot of their utility from being standard. Just like a network protocol upgrade, both sides need to agree on what the resulting behavior is. There are a few well-known interface upgrades in Go’s standard library which I will happily promote the use of (particularly the ones in io), but chances are your library isn’t standard enough to make good use of a custom interface upgrade. The downsides of sidestepping the type system are very real, and I doubt the complexity will often be worth it.

It took me a while to truly appreciate Go’s interfaces: they manage to hide a remarkable amount of complexity for such a simple idea. It’s really amazing how much power you can get from just a few well thought out interfaces and the means to convert between them, and it’s a constant reminder of how well designed Go’s standard library is.

  1. I’m sure I’m not the first to notice this phenomenon, which means that someone else has probably already given it a different name. If you know of any interesting articles about this from the literature or elsewhere on the internet, please send them my way.

  2. If for whatever reason this is impossible in your application (for instance, you rely on being able to Hijack() a http.ResponseWriter), you should be sure to have “pre-flight” runtime checks to ensure all the interface upgrades you rely on are supported.

  3. There’s actually a second system call available on Linux, splice(2), which allows the efficient transfer between an arbitrary file descriptor and a pipe. By splicing one file descriptor to one end of a pipe and another file descriptor to the other end, you can use this system call to perform arbitrary fd-to-fd copies without ever involving userspace. Unlike sendfile(2), Go doesn’t currently support this using interface upgrades, but I can’t think of a reason it couldn’t.

Why I hate HFS+

August 5, 2011

HFS+, the filesystem used on modern versions of Mac OS, is long past its prime. It’s actually a remarkable feat of engineering: the same filesystem has been extended and improved upon for over ten years despite technical limitations inherent in its original design. However, sometimes its limitations are a little too much.

I have kept a checkout of the linux 2.6 git repository on my Mac for some time now – it’s sometimes fun to go sourcediving through the kernel, even though I grok very little of it. A few months ago, however, I noticed that after a pull, git said I had unstaged changes in my working directory. This was curious since I hadn’t changed anything, but I didn’t think much of it, and after a few failed git reset --hards, I forgot about it.

Today, however, I got frustrated at git and decided to dig further into the problem. I tried everything – first I tried adding and committing the unstaged changes, but git add refused to add any of the supposedly untracked files. git reset --hard and git clean were similarly ineffective. git rm -f was the only command that I could actually get to function properly, but that put git in some odd state in which the files were both staged and not staged for deletion.

But then I looked closer at the filenames. Turns out they weren’t quite the same. The set of files that was staged for deletion were partially uppercase, while the unstaged ones were entirely lowercase:

# On branch master
# Changes to be committed:
# deleted:    include/linux/netfilter/xt_CONNMARK.h
# deleted:    include/linux/netfilter/xt_DSCP.h
# deleted:    include/linux/netfilter/xt_MARK.h
# deleted:    include/linux/netfilter/xt_RATEEST.h
# deleted:    include/linux/netfilter/xt_TCPMSS.h
# deleted:    include/linux/netfilter_ipv4/ipt_ECN.h
# deleted:    include/linux/netfilter_ipv4/ipt_TTL.h
# deleted:    include/linux/netfilter_ipv6/ip6t_HL.h
# deleted:    net/ipv4/netfilter/ipt_ECN.c
# deleted:    net/netfilter/xt_DSCP.c
# deleted:    net/netfilter/xt_HL.c
# deleted:    net/netfilter/xt_RATEEST.c
# deleted:    net/netfilter/xt_TCPMSS.c
#
# Changes not staged for commit:
# deleted:    include/linux/netfilter/xt_connmark.h
# deleted:    include/linux/netfilter/xt_dscp.h
# deleted:    include/linux/netfilter/xt_mark.h
# deleted:    include/linux/netfilter/xt_rateest.h
# deleted:    include/linux/netfilter/xt_tcpmss.h
# deleted:    include/linux/netfilter_ipv4/ipt_ecn.h
# deleted:    include/linux/netfilter_ipv4/ipt_ttl.h
# deleted:    include/linux/netfilter_ipv6/ip6t_hl.h
# deleted:    net/ipv4/netfilter/ipt_ecn.c
# deleted:    net/netfilter/xt_dscp.c
# deleted:    net/netfilter/xt_hl.c
# deleted:    net/netfilter/xt_rateest.c
# deleted:    net/netfilter/xt_tcpmss.c
#
# Untracked files:
# samples/hidraw/

Suddenly it clicked. HFS+, by default, is a case-insensitve filesystem. At some point in the last few months, the Linux repo added multiple files with different capitalizations, pretty much screwing over anyone still stuck with this miserable excuse for a filesystem.

Hey Apple. How’s that ZFS support coming along?

Screen and Agent Forwarding

July 18, 2011

GNU screen is a fantastic little program that allows you to multiplex many terminals (think tabs) in a single terminal, especially an ssh session. It also allows you to detatch from and reconnect to this session later, even after your ssh connection has died.

Learning screen isn’t hard (and there are plenty of resources on the internet to help you). Unfortunately, the most natural way to use screen is surprisingly mediocre, especially if you use ssh agent forwarding.

Starting a screen

Starting a screen session isn’t rocket science: just ssh to your server of choice and type screen. However, this isn’t a great way of doing things. First, you have to remember whether you already have a screen open on this server (if so, you need to use screen -d -r instead). Second, by ssh-ing, you’ve stared a shell that you will never use (in fact, should you ever find yourself using this shell, you’re probably going to be very sad when ssh inevitably flakes out on you).

We can fix both of these problems pretty easily. First, a quick peek at screens man page shows some interesting flags:

-d -R — Attach here and now. In detail this means: If a session is running, then reattach. If necessary detach and logout remotely first. If it was not running create it and notify the user. This is the author’s favorite.

-d -RR — Attach here and now. Whatever that means, just do it. (This one seems to do the same thing but sounds more awesome, so it’s my favorite)

-x — Attach to a not detached screen session. (Multi display mode).

-U — Run screen in UTF-8 mode. This option tells screen that your terminal sends and understands UTF-8 encoded characters. It also sets the default encoding for new windows to ‘utf8’.

We arrive at our final screen invocation, screen -UxRR. This will reattach to a screen should one already exist (but not detatching the other session), or create a new session otherwise. We also force everything into UTF-8 mode, because there’s really no reason not to.

As for the pesky superfluous shell we’ve spawned, we can turn to a few useful ssh options. ssh example.com [commandname] will run a specified command on the remote machine instead of your default shell, but it’ll complain that it “Must be connected to a terminal.” The solution here is to force ssh to allocate a TTY for the session with the -t switch.

We’re finally left with a reasonable way to ssh to a remote computer:

ssh -t example.com screen -UxRR

But what about my Agent?

ssh-agent is another really neat tool that will allow you to authenticate against your private key without typing in your key’s passphrase every time. It’s a great way to regain the ease-of-use of a passwordless key while actually having Security and other Nice Things (tm).

On its own, ssh-agent is a fantastic tool. However, combined with a feature called agent forwarding, it becomes a true powerhouse. The idea is that you allow certain hosts you’re sshed into to forward their ssh-agent requests to the agent running on your home computer. In effect, this allows you to do passwordless ssh-jumping (if, for instance, you have to connect to a gateway machine to reach an internal protected network, or if you need to supply credentials to a git server in order to commit).

Note: You should be aware of the security model of agent forwarding. In particular, anyone with root access to any machine you forward your agent to will, for the duration of your connection, be able to authenticate to other machines as you. Therefore, make sure you only forward your agent to machines that you completely trust.

However, screen is notorious for breaking agent forwarding. If a screen was created with one set of agent sockets, but that ssh connection has long since died, where should ssh-agent requests go?

Thankfully, the Internet has come up with several solutions to this problem. Unfortunately, none of them are very good—they’re generally downright hideous hacks, and in particular none of them seem to work with our improved screen invocation above (they rely on having a login shell).

Finding a non-hacky solution to this problem requires a bit of doing. First, let’s look at how ssh-agent works. If you try spawning a new ssh-agent, it’ll spit out something like this (linebreaks added for clarity)

SSH_AUTH_SOCK=/tmp/ssh-9hXYlkosFf/agent.35911;
export SSH_AUTH_SOCK;
SSH_AGENT_PID=35912;
export SSH_AGENT_PID;
echo Agent pid 35912;

When we forward our agent, essentially all that’s happening is that a socket is set up in some random location in /tmp on the remote machine that shuttles bytes back and forth between the remote machine and the UNIX socket in /tmp on our local computer. The issue for screen, then, is that this random path changes every time we connect.

Instead of keeping each screen window aware of the current socket in /tmp, as most solutions suggest, it’s easier to keep the socket in a single known location, like ~/.authsock. First, let’s tell screen about this new socket. Add the following to your .screenrc:

setenv SSH_AUTH_SOCK "$HOME/.authsock"

Now we need to ensure we keep this socket up-to-date every time we connect. For this, we symlink this path to the real socket on every connection. Normally, you might do this in your bash profile, but since we are bypassing bash entirely when we log in, that won’t work. Instead, we add the following to our ~/.ssh/rc (a file that ssh helpfully runs after every connection):

test $SSH_AUTH_SOCK && \
  ln -sf "$SSH_AUTH_SOCK" "$HOME/.authsock"

This just symlinks the appropriate socket to ~/.authsock should you be connecting with agent forwarding.

All together now

Use this to connect to remote servers:

ssh -At example.com screen -UxRR

Add the following to your .screenrc:

setenv SSH_AUTH_SOCK "$HOME/.authsock"

And the following to .ssh/rc:

test $SSH_AUTH_SOCK && \
  ln -sf "$SSH_AUTH_SOCK" "$HOME/.authsock"

MIT Mystery Hunt

January 22, 2010

This past weekend, I participated in my first ever MIT Mystery Hunt.

Mystery Hunt is an annual MIT tradition. Every year, about a thousand people split over some 40-or-so teams descend on MIT to solve some wickedly hard puzzles over the course of a weekend. And by weekend, I mean the whole weekend. None of that “sleep” stuff.

I got to Team Codex headquarters on Friday morning. The main room reminded me of Alfred Hitchcock’s Birds – literally every surface was covered with someone’s computer. I eventually found enough space for me to squat awkwardly in a chair with my laptop before we were off.

I started out on a puzzle that had me identifying bridges over the Charles River. It was plenty of fun, and between Google Street View (super-useful, even though it can be kind of sketchy too) and my somewhat patchy knowledge of Cambridge, I managed to identify a bunch of them.

Over the coming hours, I worked on trying to brute-force a particularly nasty constraints problem using Ruby (the humans beat me to it), attempting a puzzle involving frisbees and numbers that nobody ever figured out, and finding a word that corresponded to the clue “gambling top” (“teetotum,” in case you’re wondering), as well as miscellaneous legwork for other puzzles.

All in all, it was a great weekend, and I can’t wait until next year’s hunt.