Day to day stuff

Wednesday, July 12, 2017

I finally got around to writing about my hack project of last year. It was an exploration of what can be done with continuation parsers and encoders in order to implement a very fast single-copy asynchronous Thrift implementation.

Continuation parsers and encoders try to decode (read)/encode (write) their data directly from/to a network buffer. When the buffer has been fully read/written, it asks for more network buffers to continue.

Tuesday, September 13, 2016

I have been using docker in my home server since 0.4 in 2013. For me the most attractive property of docker is that it provides a way to decrease the amount of stuff one has to install on a server. I have only one server, but it has many different tasks of which some need to be rock solid (my family email) and other are experimental. Containers provide a nice way to clean up experiments. Unfortunately, docker has never been stable. I have had many fights with docker during upgrades and I have never fully understood how docker interacts with the iptables setup from my firewall (Shorewall).

Building and converting a docker image

Next step is to fetch the image into the rkt image repository. The rkt fetch command can fetch an image directly from a docker repository. However, I found no way to fetch directly from the local docker image repository. This is the work around:

Creating that statement took actually longer then I expected. Here are the highlights:

Option --dns=... copies the nameserver from you local /etc/resolv.conf to the same file inside the container. This makes DNS work inside the container also. Depending on your DNS setup, you may need to pass more --dns* options.

The --net=host gives you the easiest access to the network. If you want a bit more security, you will have to dive deeper.

I also tried to add the options --user=nobody --group=nogroup. However, that resulted in my container not starting up at all with weird error messages.

Networking

As a Shorewall user you need to add a rule to open the ports for the syncthing application in /etc/shorewall/rules and that's it. This might get a lot more hairy when you are not using host networking.

Automating startup

Modern Ubuntu's use systemd to start applications. Rkt's systemd manual seems well written but can be used as an introduction as best. Their suggestion to use systemd-run failed me:

Also, it fails to mention where to put the systemd unit file. After lots of reading (in particular the systemd section of this manual) I created /etc/systemd/system/rkt-syncthing.service with the following content:

Ending thoughts

Although I got far in a few hours there are still a few open problems.

None of the above software was available as a Ubuntu package. This results in me spending more time to keep everything up to date.

Rkt's documentation is okay but not amazing. For example, hyperlinks are missing in crucial places (see above), and in the manual page for rkt-export does not tell you how to indicate which container you want to export.

Rkt containers run from an image and store changes as a layer on top. As soon the container exists, you can not start it again with those changes. This means that everything that needs to be persisted must be external to the container. For other changes you can run rkt export to create a new image, or you rebuild your image from scratch.

Having to use Docker to build an image for rkt is a bit weird. Next step is to create the image directly with acbuild.

Updates

Thursday, March 3, 2016

In the OO world you are frowned upon if you call something object. Its time to extend this principle: don’t call your state state. This post is about why this is a bad idea and what you can do about it.

Recently we sat down to discuss the new data model for our messaging system at eBay’s Classifieds Group. One of the things we inherited from the past is the entity called conversation with a field called state. Possible values were Ok, On hold and Blocked.

So what was the problem?A field called ‘state’ almost always has a very intuitive meaning. Unfortunately, the word is so vague that the meaning can easily warp, depending on the problem at hand. I noticed this in a couple of projects: the state field started to collect more and more possible values. With more values came increasingly difficult state transitions. This lead to code that was way more messy then necessary.

For example, in our conversation entity we could introduce the state Closed to indicate that a participant wants to stop the conversation. Then we continue by adding the state Archived to indicate that the conversation should be hidden until a new messages arrive.

What can we do?The key observation is that each state value represents multiple behaviors. Think about it, what behavior is needed in each state? How do these behaviors change for each state? These questions will lead you to multiple fields that can represent the entire state of your entities.

Within a couple of minutes we found three behaviors we wanted to have for our conversations: a conversation is either visible or not (field visibility with values Displayed and Hidden), it will accept new messages or not (field acceptNew with values Accept and Reject) and we want to notify the recipient of a new message (or not) (field notifyOnNew with values Notify and Mute).Not only did our code become easier to extend and reason about, as a bonus we found a feature that would have been really hard with the old model: muting a conversation.

ConclusionDon’t call your state ‘state’, instead, think about the behavior each state represents and model that instead.

Sunday, February 14, 2016

As I run my own (secure) web and mail server I frequently have to get certificates. You could run with a selfsigned certificate, but that is not ideal; desktop browsers are very noisy about them, and mobile browsers are outright hostile. Installing certificates on a mobile phone are no fun (though, hat of for CAdroid).

Now that Letsencrypt is live, I wanted to try it out. However, Letsencrypt does not make it easy for you to keep using the same private key. This is necessary as the Android's http client pins your certificates (the one I needed anyway). Luckily Letsencrypt allows you to use a CSR that is generated outside of their tools.

Letsencrypt certificates are only valid for a short time (90 days) so automation is key. Unfortunately, openssl does not make it easy to fully automate creating CSRs, especially when you need 'Alternative Names', a requirement from Letsencrypt.

This will generate a CSR in file domain.csr with set country (C), state (ST), organization (O), organization unit (OU) and most importantly the common name (CN) and in addition two alternative names example.com and www.example.com. Thanks Andrew!

Friday, November 1, 2013

Unfortunately Apple decided to remove /usr/bin/gnutar from Maverick (Mac OSX 10.9). This is a pain because most of the tarring I do on my mac is to transfer the file to a GNU based linux (e.g. Debian/Ubuntu). Apple's bsd-tar is not compatible with gnu-tar.