Use FlatBuffers in Go

The FlatBuffers projectis awesome. In this tutorial, you’ll learn how to use it in Go.

To learn more about why we need yet another way to encode data, go read my postWhy FlatBuffers.

FlatBuffers is a serialization format from Google. It’s really fast at reading and writing your data: much quicker than JSON or XML, and often faster than Google’s other format, Protocol Buffers. It’s schema-versioned, which means your data has integrity (like in a relational database). FlatBuffers supports six programming languages: C++, C#, Go, Java, Javascript, and Python.

This post will show you how to set up FlatBuffers and then use it in a demo Go program. We’ll finish with speed measurements, because we all love micro-benchmarks!

(Full disclosure: I maintain the Go and Python ports.)

This tutorial has seven short parts:

Install the FlatBuffers compiler Write a schema definition Generate Go accessor code from the schema Install the FlatBuffers Go runtime library Write a demo Go program to encode and decode example data Write and run benchmarks Learn more and get involved

If you’d like to see all of the code in one place, I’ve put the project up at a GitHub repository.

1. Install the FlatBuffers compiler

First things first: let’s install the compiler.

The compiler is used only in development. That means you have no new system dependencies to worry about in production environments!

Installation with Homebrew on OSX

On my OSX system, I use Homebrewto manage packages. To update the Homebrew library and install FlatBuffers, run:

$ brew update$ brew install flatbuffers

Personally, I like to install the latest development version from the official Git repository:

$ brew update$ brew install flatbuffers --HEAD

If successful, you will have the flatcprogram accessible from your shell. To verify it’s installed, execute flatc:

$ flatcflatc: missing input files... Other installation methods

If you’d like to install from source, install a Windows executable, or build for Visual Studio, head over to my post Installing FlatBuffersfor more.

2. Write a schema definition

All data in FlatBuffers are defined by schemas. Schemas in FlatBuffers are plain text files, and they are similar in purpose to schemas in databases like Postgres.

We’ll work with data that make up user details for a website. It’s a trivial example, but good for an introduction. Here’s the schema:

Create a new directory for our tutorial, and place the above code in a file called myschema.fbs.

This schema defines User, which holds one user’s nameand id. The namespace for these types is users(which will be the generated Go package name). The topmost type in our object hierarchy is the root type User.

Schemas are a core part of FlatBuffers, and we’re barely scratching the surface with this one. It’s possible to have default values, vectors, objects-within-objects, enums, and more. If you’re curious, go read the documentation on the schema format.

3. Generate Go accessor code from the schema

The next step is to use the flatccompiler to generate Go code for us. It takes as input a schema file and outputs ready-to-use Go code.

In the directory with the myschema.fbsfile, run the following command:

flatc -g myschema.fbs

This will generate Go code under the directory users, which was the namespace we declared in the schema file. Here’s what the directory looks like afterwards:

$ tree.├── myschema.fbs└── users └── User.go1 directory, 2 files

One file is generated for each first class datatype. In our case, there is one file, for User.

A quick browse of users/User.goshows that there are three sections to the generated file. Here’s how to think about the different function groups:

The FlatBuffers Go runtime package is go get-able. However, because this article is a self-contained tutorial, I’m going to mangle the GOPATHenvironment variable to make installation local to this directory:

GOPATH=$(pwd) go get github.com/google/flatbuffers/go

( pwdprints the absolute path of the current directory.)

Your project directory should now have 1 file and 3 directories at the toplevel:

This function takes a FlatBuffers Builderobject and uses generated methods to write the user’s name and ID. (Note how the string value is created beforethe creation of the User object. This is needed because variable-length data are built ‘bottom to top’. I’ll write more about this in a future article.)

Reading

FlatBuffer objects are stored as byte slices, and we access the data inside using the generated functions (that the flatccompiler made for us in ./users).

Append the following code to your main.go:

// main.go part 3 of 4func ReadUser(buf []byte) (name []byte, id uint64) {// initialize a User reader from the given buffer:user := users.GetRootAsUser(buf, 0)// point the name variable to the bytes containing the encoded name:name = user.Name()// copy the user's id (since this is just a uint64):id = user.Id()return}

This function takes a byte slice as input, and initializes a FlatBuffer reader for the Usertype. It then gives us access to the name and ID values in the byte slice.

No heap allocations occur. We achieved this by using the Resetmethod on the Builderobject, and by directly using []byteslices instead of strings. We can write 1e9 / 214 ~ 4,500,000 objects per second. We can access 1e9 / 73 ~ 13,000,000 objects per second. Because this is FlatBuffers, our encoded data is schema-versioned, platform-independent, and requires no memory allocations to write or read. 7. Learn more and get involved

FlatBuffers is an active open-source project, with backing from Google. It’s Apache-licensed, and available for C++, Java, C#, Go, Python, and Javascript (with more languages on the way!).

Here are some resources to get you started:

GitHub repository Issue tracker Official documentation

I’ll be writing about FlatBuffers a lot on this blog, so stay tuned!

About the author

I'm Robert Winslow, a consulting engineer. I help teams launch great products:Learn more.