How It All Began

In the early days of Go, before 1.0 and the compatibility promise, there were weekly releases. In one of those releases, Balazs Lecz provided a new inotify package for Linux. Then Hector Chu wrote winfsnotify for Windows. Chris Howey began a package called kevent for BSD.

There was talk of designing an API that works across systems. Chris took up the challenge – in the autumn of 2011 fsnotify was born.

Fsnotify started by merging these three packages, exposing a single API. Create something on Linux and recompile it on Windows – it just works.

That’s harder than it sounds. File notifications behave differently on each operating system. Props to Chris for working through a lot of edge cases.

Continuous Feedback

From the beginning, fsnotify had users, with Rob Figueiredo testing it on OS X and later adopting it for Revel. In early 2013, I became one of those users of fsnotify.

I wanted a tool to automatically run my tests when files change, and a hackathon was the perfect opportunity to build one.

Looper watches entire subtrees, running tests for the package that changed. When you create a new package, it will automatically start watching that subdirectory as well. I was pretty pleased.

Extending Fsnotify

This recursive watcher was something I built for Looper. As I looked around, I saw I wasn’t alone. Maybe I could extract that logic into a little package other people could use and develop? There could be other useful extensions too, like the ability to coalesce duplicate events.

As I was learning, I discovered that Windows and OS X can support recursive watches natively, OS X can even coalesce duplicates!

To not take advantage of the underlying OS would be such a missed opportunity. But to do so, I couldn’t just build an extension on top of fsnotify, I would need to dig into the project itself.

Virtual Machines

This was September 2013, and I had no idea what I was getting myself into.

Before I could change fsnotify, the first obvious problem was how to test it across BSD, Linux, OS X and Windows.

Fortunately, we live in the modern era of virtual machines. With Vagrant Gopher, I got Go installed under Linux and BSD, and hand-rolled a Windows image for VMWare Fusion.

Now I could build things and go test to my hearts content. Huzzah!

Pipe Dream

I had a plan. Maybe ingenious, maybe misguided.

If I built my recursive watcher into fsnotify, later on we could optimize it away on systems that have native support.

Not only that, my generalized event processing pipeline™
could coalesce events, filter out uninteresting files, and prepare breakfast in the morning.*

I worked on it for the remainder of the year, but never merged it.

Standard Library

Before the release of Go 1.0, the inotify, winfsnotify and fsnotify packages were shuffled off to exp.

The Go Team, predominantly Russ Cox, remain interested in a file system monitoring API in the standard library. Just over a year ago, Russ released a proposal to use fsnotify to speed up the go tool itself.

It’s quite the read. To think, the Go compiler and toolchain are so fast, looking for changes with Stat is a bottleneck. There’s also mention of potentially adding a -watch flag, like having Looper built in. Awesome!

When a file or directory is created, modified or removed, the Operating System can inform any applications that wish to know. This can come in handy in many situations:

Text editors know a file has changed externally and may offer to reload it.

Applications know that there are changes to backup (Time Machine, Arq) or synchronize (Dropbox).

Search engines (Spotlight) can update their search indexes.

Static site generators (Hugo) can serve up changes as they are made.

Development tools can rebuild code or run tests automatically.

To make this work, an application simply subscribes to be notified of events.

At least it should be that simple, but unfortunately each Operating System has a different way of doing things.

“Some of you may be familiar with existing file monitoring solutions… they are a pain in the butt to use, they are complex, they are not user friendly, and they have a lot of race conditions that you have to be aware of as a developer.”Scaling Source Control at Facebook by Durham Goode

With fsnotify you have a platform-independent interface to file system notifications in the Go programming language. It tries to take the pain out of file system notifications so you can get on with writing your application.

This blog is here to provide updates on fsnotify itself and a deeper look at file system notifications for the curious. Until next time.