We recently had an opportunity to compare the garbage collection (GC) behaviour of a small Go application at different heap sizes. This application was built to migrate data away from a legacy datastore. The application contains an in-memory cache which can be configured to clear itself at different rates. This means we can look at the command during a GC cycle at various heap sizes and compare the impact of this on GC behaviour.

We found unexpected and severe whole system pauses during the garbage collector’s mark phase. This was an surprising, as Go widely advertises itself as having a low pause garbage collector. In particular we found that the expected ‘stop the world’ pause was very short, but that whole system pauses experienced during the mark phase were several orders of magnitude longer. Observing that these pauses exist is important for understanding why we see such a large performance impact from GC cycles when there are plenty of CPU resources available and the reported pauses are very short.

In this post we will detail what we saw during this experiment and discuss how this impacts the way we investigate GC performance for all of our systems going forward.

It is worth noting that the system measured here was compiled with Go version 1.10. As the GC algorithm is under constant development these effects may disappear in later releases.

Recently, I wrote a PL/PGSQL extension
which provides some basic functions for creating and altering roles,
and managing permissions. The extension was built to improve our tooling
for PostgreSQL user creation internally.
Since this has large number of external applications,
it has been released to the public under the PostgreSQL license.

This blog post is the first in a series on this extension. In it
I cover the difficulties which come with creating tooling around
utility statements in PostgreSQL as a whole, why centralising this
in user defined functions is a good idea, and what kinds of problems
we are trying to solve.

Testing has always been a must at Adjust. Serving tens of thousands of requests per second, we can’t afford tiny oopsies caused by some pointer being null when we didn’t expect it or a good old side effect in a presumably clean function. Tests are considered first-class citizens in our code base and are mandatory for any pull request that adds or changes functionality.

Just like tests, any dependency that we bring into our projects becomes a part of our code base. Since no one likes maintaining code written by some unknown dude (do you?), we try to reduce the number of third-party packages we use to an absolute minimum and prefer writing or own tailor-made libraries.

This is why we decided to stick with the minimalistic but decent testing package that Go provides as a part of its standard library.

One of the Elixir web apps we have at Adjust acts as an API gateway — it receives a request, applies authorization and/or authentication and then passes the request along to an actual requested service. As you probably know, if you’re building microservices, this approach is quite common. In our case, the API gateway also stands before a service that is responsible for generating reports. These reports are usually big json or csv blobs and sometimes they are as big as a few hundred megabytes. Because of this, downloading such a report via API gateway just to send it to a client does not sound like a good idea. Below, you can see what happened to our naïve implementation when a number of clients were trying to download sizeable reports.

In this blogpost, I’d like to describe how we’ve implemented transparent streaming of HTTP requests directly to a client.

At adjust we recently tried to replace the Go standard http library with fasthttp. Fasthttp is a low allocation, high performance HTTP library, in synthetic benchmarks the client shows a 10x performance improvement and in real systems the server has been reported to provide a 3x speedup. The service we wanted to improve makes a very large number of HTTP requests and so we were very interested in using the fasthttp client.

In the course of making the switch we encountered a number of difficulties. First the fasthttp library presents a very different interface to the programmer which must be adjusted to. Second there were a number of quirks in the implementation which made progress rather slow.

We at adjust recently started to use Elixir. We built a couple of small services using the Phoenix framework which successfully went live. In this blogpost I’d like to talk about, I’d say, the most undiscussed topic when it comes to Elixir — deployment.

In the mobile world, deep linking is a technology that launches an app and opens a specific page once a user clicks a URL on a web page or in another app. We will dive into the details of implementing deep linking for your app in this article.

Inspired by the PostgreSQL key/value data-type hstore, we developed the istore extension with
support for operators like + and aggregates like SUM for semi-structured integer-based data.

While the hstore allows arbitrary textual-data as its keys and values, in an istore document
both keys and values are represented and stored as integers. Therefore istore fits nicely in an
analytical workload. User journeys, cohort or funnel data, distributional data and many other
scenarios can be efficiently modeled and stored in PostgreSQL using istore.

The extension comes with two data types: istore and bigistore, the former having int and the
latter bigint as values; keys being int for both. This article demonstrates the efficiency of
istore and some of its applications through two examples - aggregating logs and analyzing event
funnels.

In Part III about Writing Postgres Extensions we fixed a serious bug
using LLDB debugger and completed the base36 type by using type casts.
Now it’s time to recover what we’ve actually achieved – and to do some more testing.