At this point, we’re about done discussing Omni’s push architecture! We’ve
already seen Go, the APNs data structures and connection, and the pipeline processor that manages hundreds of thousands of push notifications for
OmniFocus customers every day. All that’s left is to take a quick tour of some
other parts of the stack that support this infrastructure.

As mentioned earlier, much of Omni Sync Server is built atop FreeBSD, and the
push provider is no exception. Since Go targets FreeBSD as a “first-class
citizen” for compilation and execution, and there’s a port for the Go language,
it’s easy to get a Go toolchain up and running. Simply run pkg install lang/go
to get all the tools needed for building Go programs.

At Omni, we keep a FreeBSD system in the rotation in our automated build system.
Every few hours, it checks out the latest source for the push provider from our
internal SVN server, builds it, and archives the result. Along with the actual
Go sources for the provider, we provide a Makefile and package information, so
that the archived product is an xzipped FreeBSD package. This way, we can take
advantage of FreeBSD’s existing package management system for easy deployment
and upgrades of the push provider.

Next up, we needed a way to hang on to notification information: what clients
were registered with the provider, how they’re grouped, and some statistics
tracking. We also needed to integrate with Omni Sync Server for a staged rollout
of push: during testing, we enabled push only for some sync servers, in order to
measure the kind of extra load that push would levy on our sync system.
(Thankfully, this period was very brief, and push is enabled for every customer
now.)

All of this information was very well suited for a relational database. However,
we’re in a bit of a transitional period there as well. For older applications,
like Omni Sync Server, we already have MySQL set up and running well. For newer
code, we’re trying to use PostgreSQL wherever possible.

As a result, the push provider wound up talking to both databases! We store all
the new push-related information – things like device and group IDs, as well as
counts of notifications sent – in PostgreSQL. We also interfaced with a
read-only replica of the Omni Sync Server database for checking customers’ sync
servers. (While this might not seem like the optimal solution, we’ve been
happily using MySQL in production for a few years, and there wasn’t any sense in
potentially destabilizing everybody’s sync experience for a PostgreSQL
migration. If it ain’t broke, don’t fix it!)

The Web portion of the provider was fairly straightforward. Rather than try to
wrap Web access in Apache or nginx, or write a separate Web interface that
called a push API, we used Go’s built-in HTTP and HTML templating support to
handle all incoming HTTP requests and expose a simple but serviceable
administrative interface.

Of course, security – and future compatibility – were big concerns for push. On
top of plain HTTP, Go’s standard library also provided a flexible implementation
for handling HTTPS through the crypto/tls package. With this, we were able to
ensure that the provider passed Apple’s strict ATS requirements, and that all our connections to Apple’s push
service were encrypted as well.

In building the push provider, our engineers stood on the shoulders of giants.
Lots of hard work went into each of the components mentioned in this post, and
thanks in large part to their efficiency and utility, we were able to start work
on the provider and deploy it to our customers in less than three months. Of
course, our customers were invaluable in this process as well - hundreds of
helpful people volunteered for the OmniFocus beta program, and helped us see how
push would work together with Omni Sync Server.

As of this writing, the push provider has already sent over ten million push
notifications. We’re looking forward to the next ten!

Along the way, though, Omni’s push provider needs to do a fair bit of
preprocessing and other work for every notification it prepares to send. There’s
also the chance that we won’t actually want to send some notifications, instead
preferring to filter them out. In the current provider, we consider each
notification for:

Its bundle ID – notifications need to be sent across the right connection for
their bundle ID, and if the ID doesn’t match up with one of the current
versions of OmniFocus, we shouldn’t send it.

Logging – we want to keep some debugging information around for notifications,
just to make sure they’re all being delivered properly.

Statistics – we keep an aggregate count of all the notifications that pass
through the provider.

Associated Omni Sync Server information – while we roll out the push provider,
we want to be able to gradually enable notifications for different sync
servers, so as to measure the load that push-triggered sync levies on those
servers.

Error responses from APNs – Apple may tell us that a notification was
malformed, or our connection to APNs may drop unexpectedly.

Overall, we want an architecture that can handle all these different needs in a
unified manner. Each of these considerations needs to be its own component, but
should have an interface roughly similar to all the others, so that we can debug
them separately and also compose them easily to form the entire provider.

Luckily, Go has a wealth of documentation – not only for each API, but also in
the form of more general guides and examples. One of these discusses a
concurrency pattern that can be built entirely out of Go’s language builtins:
that of a pipeline. (Read the full article here.)

The general idea behind a Go pipeline revolves around a combination of two
important concurrency primitives: channels and goroutines. At their
simplest, a channel is a thread-safe way to pass values between two different
parts of a program, and a goroutine is a very lightweight way of running a
function on a different thread. (The actual implementation details – and
implications – are much more complex, but this should suffice for our purposes.)

If we take both channels and goroutines for granted, we can start setting up a
bunch of different pipeline components (called “stages” in the original article)
that take in a value on a channel, do some arbitrary thing with it, and push it
out on another channel. Let’s consider the example of a Notification instance
being logged out for debugging – we’ll want to just print the notification in
its entirety, then pass it down the pipeline unchanged. We might write this
pipeline component as:

With this implementation, constructing an instance of this logging pipeline
component is as simple as calling a function. It needs an existing channel as
input, but gives back a new output channel; this means that we can easily chain
multiple different components by passing the output channel from one function to
the input of another.

We run the “pump” for this component as a goroutine, so that it’s always running
alongside the rest of our program. Inside, we use the range of the input
channel, so that the goroutine blocks until a notification comes through. When
the input channel closes, this for loop over the input channel terminates. At
that point, we’ll close the output channel too, signaling down the pipeline that
we’re done handling notifications.

For the push provider’s use, we can do a bunch of different things in each of
these components – and logging is only the simplest! The provider itself is
structured as a pipeline with nearly a dozen components from start to end:

The logging component looks very similar to the above, with just a little bit
more configurability about where to send logs (and with what severity).

The statistics component is also similar to the above, but instead of logging
out a message, it increments a few internal counters depending on the contents
of the notification.

The OSS component uses the UserInfo map [mentioned previously](TODO%20link%20to
%20%20part%202), which we populate with the sender’s OSS username when building some
Notification instances. If we need to drop a notification – perhaps because
the sending user is on a sync server that’s under heavy load – we can simply
refuse to pass it along the output channel from this component.

Even the persistent connection to APNs is handled with a pair of these pipeline
components. The first takes in notifications, sends them over to APNs, and
forwards them unconditionally; the second then buffers the sent notifications
briefly to await an error, then forwards them in turn.

Keep in mind, too, that the pipeline components can have more than one input or
output channel! Omni’s provider also includes a pipeline multiplexer and
demultiplexer. These components are used to split up the notification stream
(based on the associated bundle ID) for transmission to APNs, then rejoin the
stream later after the APNs components pass the sent notifications through.

This sort of pipeline architecture is how the provider can handle Omni’s volumes
of push notifications. At its peak, our single push provider instance transmits
close to 500 notifications every minute – while still using only a few percent
of one server’s CPU.

While Go makes this sort of concurrency quick to write and efficiently scalable,
it’s not the only piece involved in the puzzle. Next time, we’ll discuss some
other technologies involved in the provider stack, including the backing
database used to store device registrations.

Now that we’ve converted our notification data into a format that’s suitable for
sending to Apple, our fledgling push provider needs a connection into APNs in
order to send that data. Once again, Apple’s documentation comes to our rescue:
the chapter “Provider Communication with Apple Push Notification
Service”
describes how to manage connections to the push notification service.

Among the important bits in this chapter:

When connecting, the provider needs the SSL certificate from the Member Center
available for authenticating itself to Apple.

The provider should maintain a persistent connection to APNs, rather than
connecting and disconnecting for each notification. (At higher volumes, Apple
might even consider this behavior to be a DDOS attack!)

There are two different environments for push notifications:
development and production. A connection is specific to one app’s
bundle ID, and therefore (at least in Omni’s bundle ID configuration) specific
to one of these environments.

To handle all this, we need to build a little flexibility into the provider. We
turned to the open-source gcfg library, which
can be imported into a Go project simply by running

go get "code.google.com/p/gcfg"

and then importing that same URL at the head of a Go file. With this library,
we can define a configuration file that tells the provider about what APNs
host(s) it should connect to, and what certificates it should use along the way.

Let’s take a look first at the Go struct that the provider reads its APNs
configuration into:

This struct expresses a single connection to APNs. It defines the domain
we’ll connect to, letting us switch between sandbox and production environments
by changing the host we contact. It also lets us tell the provider where its
certificates are, so that it can establish a secure connection and identify
itself to APNs, all in one swoop.

For OmniFocus, though, we have multiple bundle IDs – one for the Universal
version of the app, and one for the iPhone-only variant – so we’ll need a way to
connect to APNs multiple times. To accomplish this, we parse something slightly
more complex out of the configuration file:

Instead of just reading a single instance of the ConnConfig struct, we’ll read
an entire map of them, keyed by strings. The gcfg package uses maps to
represent configuration subsections – instead of just specifying a single
apns section, we can specify a bunch, each with its own key:

This way, parsing a configuration file can return one ConnConfig struct for
each bundle ID that we’ll use to connect. The provider’s connection code can
then iterate over these structs, establishing multiple connections along the
way. For each connection, that code is fairly straightforward:

Once this connection is established, the provider can hang on to this tls.Conn
pointer as long as it needs to, sending multiple notifications – in the form of
the frames described in the previous post – when changes
occur in OmniFocus. In the next post, we’ll take a look at how the provider
manages all the notifications it needs to send across these connections.

After researching languages and choosing Go to implement a push provider for
OmniFocus, we needed to get started writing code. Apple has set out the
interface for a provider in the chapter “Provider Communication with Apple Push
Notification
Service;”
our task then became to write a server that could speak in the binary format
APNs expected.

Every notification delivered to Apple is built of a series of items. Each
item details one part of the notification, such as its JSON payload or metadata
like its priority or expiration date. Items have a common format: each begins
with one byte giving the item type, then two bytes giving its total length. All
the actual information in the item follows that three-byte “header.”

This is pretty straightforward to represent in Go: we just need a struct whose
layout matches the expected format of an item. Let’s write one now:

type Item struct {
ID byte
dataLength uint16
data []byte
}

From this basic definition, we can represent all the different kinds of item
that might appear in an APNs notification. Since the data in an item is stored
in its encoded (byte slice) form, we can also write this item to a binary buffer
for transmission to Apple very easily:

We can also take advantage of Go’s iota
construct inside constant declarations
to define names for each kind of item that Apple documents. Since the items are
1-indexed, we simply define the first as 1 + iota and let the language go from
there:

While some items are a little more complex than this one-liner, in general,
creating any single item is simply a matter of parsing its data into a byte
slice and returning one of these structs. We can then begin composing these
items into larger pieces of data for more convenient use.

The next step is to build an entire frame for transmission to Apple. A frame
is simply a list of items, all serialized next to each other and wrapped with a
few more bytes’ worth of header data. We can take a similar approach for
building frames as we did for items: create a struct that contains a slice of
Item structs, then implement a Bytes() function for that struct that
recursively encodes all the contained Items and adds the required header data.

Strictly speaking, this is all our push provider needs to start sending
notifications! We can construct frames full of items, serialize them, and send
them to Apple. However, we’ll probably want a little more metadata internally to
help our notifications along. At the very least, we’ll need to know the bundle
ID associated with each notification – since the connection we establish to
APNs is specific to one app’s bundle ID, and we’re providing push services for
multiple apps, we’ll need a way to distinguish between different notifications
so we know which connection to use.

To handle this, we wrap up our Frame in one more top-level struct: the
Notification. All this struct does is provide information that our provider
tracks internally – it doesn’t send anything extra to Apple. We define it as:

This last field is a little quirky: it maps strings to interface{}, which is
Go’s “anything” type. With this field, the provider yields a bit to Objective-C
– we can attach any extra bits of data that we want internally in this
UserInfo map, then pull them back out later. We’ll see an example of how this
is used in a future post.

At this point, we’re ready to start talking to Apple! We have representations of
all the different pieces of data that the APNs binary interface expects, at
least for sending notifications. In the next post, we’ll explore how the push
provider connects to the APNs gateway.

Here at the Omni Group, we have a long history of writing code in Objective-C.
All of the apps we currently sell are written in Objective-C, and those of you
that follow our public frameworks will
know they’re entirely Objective-C as well.

However, we recently faced a new challenge with our decision to bring
Push-Triggered Sync to OmniFocus on Mac and iOS. Apple’s architecture for
sending push notifications requires that we use a server-side component called a
provider. And while Objective-C was our forte, it was ill-suited for server-side
development, since almost none of Omni’s servers run OS X.

Thus, our attention turned to building a provider that could handle the large
existing OmniFocus customer base, along with the specific traffic patterns the
app generates. (For example, the Omni Sync
Server often sees large traffic spikes early
Monday mornings, as OmniFocus users get started on their work weeks.) What’s
more, we wanted this provider to fit well in our existing server setup, which is
gradually moving towards using FreeBSD.

With these constraints in mind, we settled on the Go
language for implementing a push provider. Go had several
properties that we felt made it a good fit for this feature:

It was explicitly targeted at system-level services, rather than our usual
focus on apps with rich user interfaces. In particular, Go’s multitasking
features (such as first-class functions and goroutines) made it well suited
for building a parallelizable server.

It has a strong standard library, especially around networking and
manipulating arrays of bytes directly — two especially relevant topics for
interacting with Apple’s binary notifications interface across the Internet.

It is a relatively young language, and so is able to take advantage of recent
advancements and best practices in language design. On the flip side, it’s not
too young: with Go 1.4, the language has reached a level of maturity
sufficient to freeze the API, providing a certain amount of stability in Go
projects.

It’s a compiled language with FreeBSD as a “first-class” targeted platform,
giving us confidence that we could deploy a provider binary across our hosting
infrastructure without needing a custom environment or runtime installed up
front.

After settling on Go, we also considered various open-source packages for
speeding up early development. And while we did wind up using a few (for tasks
like MySQL and PostgreSQL database access, as well as logging), existing
frameworks for communicating with Apple’s push service (APNs) didn’t quite look
as fully-formed as we would’ve liked.

With that, the stage was set: in Go, build an internal framework for
communicating with APNs, then use it in a provider daemon that could power push
notifications for the hundreds of thousands of customers that rely on OmniFocus
every day. We’ll discuss more about specific implementation details next.