Why I hate HFS+

August 5, 2011

HFS+, the filesystem used on modern versions of Mac OS, is long past its prime. It’s actually a remarkable feat of engineering: the same filesystem has been extended and improved upon for over ten years despite technical limitations inherent in its original design. However, sometimes its limitations are a little too much.

I have kept a checkout of the linux 2.6 git repository on my Mac for some time now – it’s sometimes fun to go sourcediving through the kernel, even though I grok very little of it. A few months ago, however, I noticed that after a pull, git said I had unstaged changes in my working directory. This was curious since I hadn’t changed anything, but I didn’t think much of it, and after a few failed git reset --hards, I forgot about it.

Today, however, I got frustrated at git and decided to dig further into the problem. I tried everything – first I tried adding and committing the unstaged changes, but git add refused to add any of the supposedly untracked files. git reset --hard and git clean were similarly ineffective. git rm -f was the only command that I could actually get to function properly, but that put git in some odd state in which the files were both staged and not staged for deletion.

But then I looked closer at the filenames. Turns out they weren’t quite the same. The set of files that was staged for deletion were partially uppercase, while the unstaged ones were entirely lowercase:

# On branch master
# Changes to be committed:
# deleted:    include/linux/netfilter/xt_CONNMARK.h
# deleted:    include/linux/netfilter/xt_DSCP.h
# deleted:    include/linux/netfilter/xt_MARK.h
# deleted:    include/linux/netfilter/xt_RATEEST.h
# deleted:    include/linux/netfilter/xt_TCPMSS.h
# deleted:    include/linux/netfilter_ipv4/ipt_ECN.h
# deleted:    include/linux/netfilter_ipv4/ipt_TTL.h
# deleted:    include/linux/netfilter_ipv6/ip6t_HL.h
# deleted:    net/ipv4/netfilter/ipt_ECN.c
# deleted:    net/netfilter/xt_DSCP.c
# deleted:    net/netfilter/xt_HL.c
# deleted:    net/netfilter/xt_RATEEST.c
# deleted:    net/netfilter/xt_TCPMSS.c
# Changes not staged for commit:
# deleted:    include/linux/netfilter/xt_connmark.h
# deleted:    include/linux/netfilter/xt_dscp.h
# deleted:    include/linux/netfilter/xt_mark.h
# deleted:    include/linux/netfilter/xt_rateest.h
# deleted:    include/linux/netfilter/xt_tcpmss.h
# deleted:    include/linux/netfilter_ipv4/ipt_ecn.h
# deleted:    include/linux/netfilter_ipv4/ipt_ttl.h
# deleted:    include/linux/netfilter_ipv6/ip6t_hl.h
# deleted:    net/ipv4/netfilter/ipt_ecn.c
# deleted:    net/netfilter/xt_dscp.c
# deleted:    net/netfilter/xt_hl.c
# deleted:    net/netfilter/xt_rateest.c
# deleted:    net/netfilter/xt_tcpmss.c
# Untracked files:
# samples/hidraw/

Suddenly it clicked. HFS+, by default, is a case-insensitve filesystem. At some point in the last few months, the Linux repo added multiple files with different capitalizations, pretty much screwing over anyone still stuck with this miserable excuse for a filesystem.

Hey Apple. How’s that ZFS support coming along?

Screen and Agent Forwarding

July 18, 2011

GNU screen is a fantastic little program that allows you to multiplex many terminals (think tabs) in a single terminal, especially an ssh session. It also allows you to detatch from and reconnect to this session later, even after your ssh connection has died.

Learning screen isn’t hard (and there are plenty of resources on the internet to help you). Unfortunately, the most natural way to use screen is surprisingly mediocre, especially if you use ssh agent forwarding.

Starting a screen

Starting a screen session isn’t rocket science: just ssh to your server of choice and type screen. However, this isn’t a great way of doing things. First, you have to remember whether you already have a screen open on this server (if so, you need to use screen -d -r instead). Second, by ssh-ing, you’ve stared a shell that you will never use (in fact, should you ever find yourself using this shell, you’re probably going to be very sad when ssh inevitably flakes out on you).

We can fix both of these problems pretty easily. First, a quick peek at screens man page shows some interesting flags:

-d -R — Attach here and now. In detail this means: If a session is running, then reattach. If necessary detach and logout remotely first. If it was not running create it and notify the user. This is the author’s favorite.

-d -RR — Attach here and now. Whatever that means, just do it. (This one seems to do the same thing but sounds more awesome, so it’s my favorite)

-x — Attach to a not detached screen session. (Multi display mode).

-U — Run screen in UTF-8 mode. This option tells screen that your terminal sends and understands UTF-8 encoded characters. It also sets the default encoding for new windows to ‘utf8’.

We arrive at our final screen invocation, screen -UxRR. This will reattach to a screen should one already exist (but not detatching the other session), or create a new session otherwise. We also force everything into UTF-8 mode, because there’s really no reason not to.

As for the pesky superfluous shell we’ve spawned, we can turn to a few useful ssh options. ssh example.com [commandname] will run a specified command on the remote machine instead of your default shell, but it’ll complain that it “Must be connected to a terminal.” The solution here is to force ssh to allocate a TTY for the session with the -t switch.

We’re finally left with a reasonable way to ssh to a remote computer:

ssh -t example.com screen -UxRR

But what about my Agent?

ssh-agent is another really neat tool that will allow you to authenticate against your private key without typing in your key’s passphrase every time. It’s a great way to regain the ease-of-use of a passwordless key while actually having Security and other Nice Things (tm).

On its own, ssh-agent is a fantastic tool. However, combined with a feature called agent forwarding, it becomes a true powerhouse. The idea is that you allow certain hosts you’re sshed into to forward their ssh-agent requests to the agent running on your home computer. In effect, this allows you to do passwordless ssh-jumping (if, for instance, you have to connect to a gateway machine to reach an internal protected network, or if you need to supply credentials to a git server in order to commit).

Note: You should be aware of the security model of agent forwarding. In particular, anyone with root access to any machine you forward your agent to will, for the duration of your connection, be able to authenticate to other machines as you. Therefore, make sure you only forward your agent to machines that you completely trust.

However, screen is notorious for breaking agent forwarding. If a screen was created with one set of agent sockets, but that ssh connection has long since died, where should ssh-agent requests go?

Thankfully, the Internet has come up with several solutions to this problem. Unfortunately, none of them are very good—they’re generally downright hideous hacks, and in particular none of them seem to work with our improved screen invocation above (they rely on having a login shell).

Finding a non-hacky solution to this problem requires a bit of doing. First, let’s look at how ssh-agent works. If you try spawning a new ssh-agent, it’ll spit out something like this (linebreaks added for clarity)

echo Agent pid 35912;

When we forward our agent, essentially all that’s happening is that a socket is set up in some random location in /tmp on the remote machine that shuttles bytes back and forth between the remote machine and the UNIX socket in /tmp on our local computer. The issue for screen, then, is that this random path changes every time we connect.

Instead of keeping each screen window aware of the current socket in /tmp, as most solutions suggest, it’s easier to keep the socket in a single known location, like ~/.authsock. First, let’s tell screen about this new socket. Add the following to your .screenrc:

setenv SSH_AUTH_SOCK "$HOME/.authsock"

Now we need to ensure we keep this socket up-to-date every time we connect. For this, we symlink this path to the real socket on every connection. Normally, you might do this in your bash profile, but since we are bypassing bash entirely when we log in, that won’t work. Instead, we add the following to our ~/.ssh/rc (a file that ssh helpfully runs after every connection):

test $SSH_AUTH_SOCK && \
  ln -sf "$SSH_AUTH_SOCK" "$HOME/.authsock"

This just symlinks the appropriate socket to ~/.authsock should you be connecting with agent forwarding.

All together now

Use this to connect to remote servers:

ssh -At example.com screen -UxRR

Add the following to your .screenrc:

setenv SSH_AUTH_SOCK "$HOME/.authsock"

And the following to .ssh/rc:

test $SSH_AUTH_SOCK && \
  ln -sf "$SSH_AUTH_SOCK" "$HOME/.authsock"

MIT Mystery Hunt

January 22, 2010

This past weekend, I participated in my first ever MIT Mystery Hunt.

Mystery Hunt is an annual MIT tradition. Every year, about a thousand people split over some 40-or-so teams descend on MIT to solve some wickedly hard puzzles over the course of a weekend. And by weekend, I mean the whole weekend. None of that “sleep” stuff.

I got to Team Codex headquarters on Friday morning. The main room reminded me of Alfred Hitchcock’s Birds – literally every surface was covered with someone’s computer. I eventually found enough space for me to squat awkwardly in a chair with my laptop before we were off.

I started out on a puzzle that had me identifying bridges over the Charles River. It was plenty of fun, and between Google Street View (super-useful, even though it can be kind of sketchy too) and my somewhat patchy knowledge of Cambridge, I managed to identify a bunch of them.

Over the coming hours, I worked on trying to brute-force a particularly nasty constraints problem using Ruby (the humans beat me to it), attempting a puzzle involving frisbees and numbers that nobody ever figured out, and finding a word that corresponded to the clue “gambling top” (“teetotum,” in case you’re wondering), as well as miscellaneous legwork for other puzzles.

All in all, it was a great weekend, and I can’t wait until next year’s hunt.