20 Oct 2016, 08:58

There are a lot of different solutions when it comes to collecting metrics, I found myself happy with this hybrid solution.

Telegraf is an agent written in Go for collecting metrics from the system it’s running on.
It’s developed by Influxdata the people behind InfluxDB, but Telegraf has a lot of outputs plugins and can be used without InfluxDB.
Many different platform (FreeBSD, Linux, x86, Arm …) are offered and only one single static binary (Thanks to Golang) is needed to deploy an agent.

Install Prometheus

Prometheus is a special beast in the monitoring world, the agents are not connecting to the server, it’s the opposite the server is scrapping the agents.
In this config we are creating a job called telegraf to be scrapped every 10s connecting to mynode host on port 9126.

That’s all you need to run a Prometheus server, start it by specifying a path to store the metrics and the path of the config file:

28 Sep 2016, 13:15

If you are following Go development you probably know that:
Go 1.7 moves the golang.org/x/net/context package into the standard library as context, Yeah !
Unfortunately it won’t work for everything, I’ve spent some time understanding this one.

For example if you are using gRPC you can hit this problem, here is an interface generated by gRPC:

The compiler is complaining about wrong types for the context argument.
Problem is gRPC generated code is importing context as golang.org/x/net/context, that’s the only way it remains compatible between Go 1.6 & Go 1.7.

So a quick solution is to import context in your own code to use the old path:

// we have to use the old context path here for gRPC compat see https://github.com/grpc/grpc-go/issues/711
context "golang.org/x/net/context"

22 Sep 2016, 15:39

Go Mobile can generate native framework for iOS and Android using Go code, I was curious what could be achieved with it.
Most tutorials are Hello world and I wanted to test it with real code.
You can use it to generate a full app only using Go code, but I’m only interested by the bindings part (SDK applications), using a native ObjC/Swift app calling Go code.

So the first thing to do is to write some Go wrapper like you would do in ObjC to call some C++ code but first:

Functions must return either no results, one result, or two results where the type of the second is the built-in error type.
So far no major complication, but also note that you can’t return slices, that could be a major pain, there are some workaround solutions like go-mobile-collection that can generate an API to operate on slice.
You also have to respect some naming for your constructor like New...().

At the end your wrapper won’t look very Goish but it’s what it takes for it to be translated.

It’s performing very well, on my iPhone 6, around 4,000 queries per second to test a position in a small fence, to 60,000 queries in a hit miss (calling Go overhead is not that big), while using a ridiculously small amount of memory.

I’ve made a demo iOS app where you can hit the map and it tells you in which fence you are.

Remember this is running locally on your phone without any network access (but the map), the geo computation and the Polygons are returned by Go code.

Until now I had to maintain libraries in both languages, Go mobile is a nice alternative!

20 Sep 2016, 17:40

I’ve been using gRPC at work and in several personal projects for months and happy with it, but when it comes to load balancing gRPC does not come with batteries included.

For a long time the only document was the Load Balancing draft in the gRPC repo, the clients should implement a Picker interface to know about the servers, so the pooling and controling the load were handled by the clients.
HTTP/2 was new and most of the reverse proxies implementations were not capable of load balancing gRPC HTTP2 frames, the only solution was to use a TCP load balancer, generating errors, improper and weird behaviours for the clients.

22 Jul 2016, 19:11

I have found the right setup to stream in 1080p from a Raspberry Pi using the camera to TwitchTV while injecting audio on the fly!

Create an account on Twitch and grab you stream key in the Dashboard.

This stream.sh script will create a FIFO start rtl_fm at freq 162.550M to listen to Canada weather bulletin (use your local NOAA channel) and inject and encode the audio to the existing h264 stream from the camera then stream it to twitch using rtmp.

03 Jul 2016, 08:15

I’ve been using Jetbrains editor (the free Idea community edition) or Pycharm with the Go plugins and very happy with this setup, the editor is providing some realtime linting but I was missing gometalinter.

08 Mar 2016, 11:23

If you read this blog, you know I’ve recently released a project called regionagogo, a geo shape lookup database, described in this blogpost.

It uses the current Go S2 implementation, which is not yet as complete as the C++ implementation, for example the region coverer of a shape does not really compute cell around the shape but around the bounding box instead.

Using the shape of the polygon makes the covered cells more precise and smaller, resulting at the end to less PIP tests which are costly.

The same coverage with Go S2 would have returned 8 big cells of the same size, covering over regions.

I’ve created regionagogogen a quick and dirty command line program for OSX that takes a GeoJSON file containing your regions and then compute the database using the S2 C++ implementation, it’s for OSX only cause I’m using ObjC as a bridge to C++ which I don’t know enough.

Also note that regionagogogen includes an S2 port that works on iOS/OSX and some ObjC geo helper classes.

18 Feb 2016, 17:16

On a previous post, I’ve described how to use the S2 geo library to create a fast geo database, but it was to store locations (points) and only to perform range queries, a complete geo database would have regions/polygons queries.

Looking for a solution

I had this need: querying for the countries or subregions of hundreds of coordinates per second, without relying on an external service.

One solution, using my previous technique, could have been to store every cities in the world and then perform a proximity query around my point to get the closest cities, but it works only in populated area and it’s only an approximation.

I looked into others solutions, there is some smart ideas using UTF-grid, but it’s a client side solution and also an approximation tied to the resolution of the computed grid.

S2 to the rescue

S2 cells have some nice properties, they are segments on the Hilbert curve, expressed as range of uint64, so I had the intuition the problem to perform fast region lookup could be simplified as find all mathematical segments containing my location expressed as an uint64.

Using a Segment Tree datastructure, I first tried an in memory engine, using Natural Earth Data, loading the whole world countries shapes into S2 loops (a Loop represents a simple spherical polygon), transforming then into cells using the region coverer, it returns cells of different levels, add them to the segment tree.

To query, simply tranform the location into an S2 Cell (level 30) and perform a stubbing query that intersects the segments, every segments crossed are cells that covered a part of a Loop.
It will reduce the problem to test a few Loop vs thousands of them, finally perform ContainsPoint against the found loops cause the point could be inside the Cell but not inside the Loop itself.

Et voilà! It works!

The segment tree structure itself is very low on memory, the loops/polygons data could be stored on disk and loaded on requests, I’ve tested a second implementation using LevelDB using this technique.

Region a gogo

As a demonstration here is a working microservice called regionagogo, simply returning the country & state for a given location.
It loads geo data for the whole world and answers to HTTP queries using small amout of memory.

Note that it performs really well but can be improved a lot, for example the actual Go S2 implementation is still using Rect boxing around loops, that’s why regionagogo is using a data file so it can be generated from the C++ version.

Future

This technique seems to work well for stubbing queries, region queries, geofencing …
It can be a solid foundation to create a flexible and simple geo database.

26 Jan 2016, 14:01

Six months ago, I wrote on this blog about Geohashes and LevelDB with Go, to create a fast geo database.
This post is very similar as it works the same way but replacing GeoHashes with Google S2 library for better performances.

For the storage this post will stay agnostic to avoid any troll, but it applies to any Key Value storages: LevelDB/RocksDB, LMDB, Redis…
I personnaly use BoltDB and gtreap for my experimentations.

This post will focus on Go usage but can be applied to any languages.

Or skip to the images below for visual explanations.

Why not Geohash?

Geohash is a great solution to perform geo coordinates queries but the way it works can sometimes be an issue with your data.

Remember geohashes are cells of 12 different widths from 5000km to 3.7cm, when you perform a lookup around a position, if your position is close to a cell’s edge you could miss some points from the adjacent cell, that’s why you have to query for the 8 neightbour cells, it means 9 range queries into your DB to find all the closest points from your location.

If your lookup does not fit in level 4 39km by 19.5km, the next level is 156km by 156km!

The query is not performed around your coordinates, you search for the cell you are in then you query for the adjacent cells at the same level/radius, based on your needs, it means it works very approximately and you can only perform ‘circles’ lookup around the cell you are in.

The most precise geohash needs 12 bytes storage.

-90 +90 and +180 -180, -0 +0 are not sides by sides prefixes.

Why S2?

S2 cells have a level ranging from 30 ~0.7cm² to 0 ~85,000,000km².
S2 cells are encoded on an uint64, easy to store.

The main advantage is the region coverer algorithm, give it a region and the maximum number of cells you want, S2 will return some cells at different levels that cover the region you asked for, remember one cell corresponds to a range lookup you’ll have to perform in your database.

The coverage is more accurate it means less read from the DB, less objects unmarshalling…

Real world study

We want to query for objects inside Paris city limits using a rectangle:

Using level 5 we can’t fit the left part of the city.
We could add 3 cells (12 total DB queries ) on the left but most algorithms will zoom out to level 4.

But now we are querying for the whole region.

Using s2 asking for 9 cells using a rectangle around the city limits.

The zones queried by Geohash in pink and S2 in green.

Example S2 storage

Let’s say we want to store every cities in the world and perform a lookup to find the closest cities around, first we need to compute the CellId for each cities.

RegionCoverer will return at most 8 cells (in this case 7 cells: 4 8, 1 7, 1 9, 1 10) that is guaranteed to cover the given region, it means we may have to exclude cities that were NOT in our rect, with func (Rect) ContainsLatLng(LatLng).

Congrats we have a working geo db.

S2 can do more with complex shapes like Polygons and includes a lot of tools to compute distances, areas, intersections between shapes…

11 Jan 2016, 09:33

I’m happy to announce a new side project: MyAPRS, a modern iOS APRS application, for radio amateur enthusiasts.

I’ve already mentioned APRS on this blog, it will mainly be useful for radio amateurs but can be interesting to RTL-SDR listeners too.

The application is built around LevelDB and geo hashing as mentioned in this post blog, it’s a lot faster than using SQLite especially on iOS, SatSat is still using the SQLite cities lookup and you can compare it’s terribly slow.
It receives weather data & APRS data from APRS-IS connections, then decodes and stores the packets into LevelDB for indexation.

MyAPRS has some extra features over other applications, like transceiver models detection, so it can highlight all C4FM/Fusion users in your area.
It receives the latest 10mn packets for your area, helped by a small Golang API.

The first release is simple but works very well to discover repeaters and amateurs around you or a region you plan to visit, I’ll add position sharing later: due to the way APRS-IS works it’s the responsability of the developer to provide passwords & check for the amateur license of the users (even if the algorithm is known publicly…).