Geeking about

I’ve got back into reading books a bit more this year, and one of the key things I’ve done is decide not to read the internet at certain times – over breakfast being one of the key times. And I had a 2 week holiday in Scotland doing a litle light walking and a lot of reading :)

Anyway, here are the books by theme:

Pregnancy

My partner became pregnant in 2015 so we of course read quite a bit about the process.

“Bump – How to make, grow and birth a baby” and “The Food of Love” (about breastfeeding) – both by the awesome Kate Evans. She has a lot of sound advice (mostly backed up by evidence), a lovely voice and some very funny cartoons. It’s the only book I’ve seen that explicitly acknowledges lesbian couples throughout the book. And the voice is by turns caring, witty and always supportive. Highly recommended if you’re heading that way or thinking about heading that way.

Bumpology is the book when you’re getting tired or bewildered by all the conflicting advice you’ll read, be told etc. Written by a journalist for the New Scientist, this book is all about the evidence. And refreshingly she presents the numbers involved in studies and translates risk into phrases like “so it would require 5000 woman to stop doing X to prevent one still birth”. Very nice.

Politics, History etc

The Joy of Tax by Richard Murphy takes a fairly radical look at taxation and the part it makes in the modern state and economy. He explains that governments start by spending money and putting it into the economy, and tax is then used to regulate the money supply, enourage or discourage behaviour and a few other broad purposes. Once you take on that view, it changes a lot of how you look at government, deficits and the rest. A lot of his ideas have been used to make Corbynomics and I’m very happy to see the terms of the economic debate being greatly widened. Well worth a read for a very different perspective to the usual neoliberal one.

The Great War for Civilisation by Robert Fisk is an epic history of the Middle East. Fisk pulls no punches, and consequently the read borders on the traumatic for periods. I put it down after one section detailing how one state had tortured various dissidents over time. I’ll pick it up again, but it’s a hard read at times.

Science, Tech, Work

Irrationality by Stuart Sutherland is a classic about the many ways the brain makes shortcuts that often serve as well, but in the modern world often lead us astray. It’s a fascinating read, but you’ll probably remain just as irrational after reading it.

2 Scoops of Django 1.8 is a collection of best practices for using the Django web framework, written by a couple who have been in the core of the Django community for many years. Nothing earth shattering in there, but lots of solid stuff. Generally, I’d start with their recommendations and only change if there is a good reason.

Refactoring by Martin Fowler is one of those classics where the word has become part of the lexicon and then used and abused by all and sundry. It is interesting to read the definition as “changing the internals of the code without changing the external behaviour”. So the idea is to get all the tests passing, and then refactor the code to improve it. You may well improve it in a way that will make the next feature easy, but you’re not making the change at that point. As Kent Beck puts it “for each desired change, make the change easy (warning: this may be hard), then make the easy change”. The refactorings presented are generally pretty low level, leading to a fine grained style of refactoring.

SMACSS is a way to organise your CSS. God knows I need a better way to do this, though since reading it I’ve not done much CSS. I shall try to pick it up again when I do get back there.

Rolling Rocks Downhill and The Phoenix Project are both novels about project management in an IT setting – the first more focussed on software development, the second more on Ops and DevOps. I’m not sure they’re the highest forms of literature, but they’re surprisingly good page turners, and have plenty of good ideas for making projects run smoother.

Never Check Email in the Morning was recommended on a blog I respect. The style is relentlessly positive American business which can be a bit much. But the content is generally sound, so if you’re struggling to manage to keep on top of your work, you may well get some value out of this book.

Fiction

This year I read the first 3 of the Enders series – Enders Game, Speaker for the Dead, Xenocide. Fantastic sci-fi that gets beyond the initial premise into deep relationships, well formed characters and difficult quandaries. The first book has less of that, but it’s really the kicking off point for some sudden changes of gear. I won’t spoil things by talking about the plot.

The Endless War is another classic sci-fi, looking at long wars and adapting back to society afterwards. I enjoyed it but it’s not one of the greatest I’ve read. Ancillary Justice is a more recent sci-fi that has a different take on the world with a fairly amoral society producing a remnant of a warship looking for revenge. It feels pretty fresh.

The Circle by Dave Eggers is a near future distopia involving some sort of combination of Google and Facebook called The Circle. Quite disturbing, if a little heavy handed in places. “Sharing is Caring.”

I also came back (again) to Neil Gaiman, re-reading the lovely American Gods, really enjoying the prequel “Sandman Overture” – fantastic artwork and ideas – and also the beautifully illustrated “Sleeper and the Spindle”.

Nothing buddhist in 2015, which is unusual for me. We’ll see how I feel in 2016, though with a baby arriving imminently I may end up with less reading time …

TTY console

One final thing that’s pretty much unusable out of the box are the TTY consoles you get from pressing, eg, CTRL+ALT+F1. To change their font, you’ll want to do a:

sudo dpkg-reconfigure console-setup

I picked the VGA font in 32x16, and it looks fine, if somewhat retro.

Touchpad Middle Button

The touchpad worked pretty well out of the box. Two finger scrolling was smooth, but …

I really like using the middle button in Linux, for pasting selected text, opening links in a new browser tab, closing browser tabs … But I couldn’t find a way to make it work. Various people mentioned that the top right corner of the touchpad/clickpad should work, but it didn’t for me. Eventually I worked out how to do it with synclient. First I had to look up all the sizes of stuff. It’s a long list, but here are the relevant results:

It’s just a little thing, but I like to see counts of messages in certain categories. As notmuch provides a count command, it is simple to write a script that will provide you with the counts you want. Here’s one I prepared earlier:

#!/bin/sh
# I'm interested in the thread count rather than individual messages
NMC="/usr/bin/notmuch count --output=threads"
# I break it down by total, how many are flagged and how many are unread
# This function produces a single line
function tag_report(){
TAG=$1
TOTAL_COUNT=$($NMC tag:$TAG)
FLAGGED_COUNT=$($NMC tag:$TAG AND tag:flagged)
UNREAD_COUNT=$($NMC tag:$TAG AND tag:unread)
echo "$TAG: total $TOTAL_COUNT flagged $FLAGGED_COUNT unread $UNREAD_COUNT"
}
# then these are the tags I'm interested in, so print the reports
# one line per tag
function full_report(){
tag_report inbox
tag_report action
tag_report waiting
tag_report readlater
}
# finally, pipe through column to make the results line up nicely
full_report | column -t

I’ve chosen to do this by having a small tmux window above my mail client, using watch to refresh. The command I use is:

watch -t -n 300 bin/mailreport

The -t means don’t bother showing the time (saving a line of output) and the -n 300 means only run the report once every 5 minutes. The output (currently) looks like:

notmuch is somewhat limited in the choice of headers it processes, despite that being an oft requested feature. However afew parses the entire email itself so it can do what it likes.

The recently added HeaderMatchingFilter will match a regular expression you provide against an arbitrary header you specify. The text matched can be used as the tag. A simple version of the ListMailsFilter could be implemented as:

I love the way gmail uses labels and I loved using them in sup. notmuch has tags for the same purpose so I want to use them extensively, and to have most tags added automatically for me.

notmuch new mail workflow

There are several ways to do this, but the way I’ve done it is:

offlineimap runs every X minutes

the offlineimap postsynchook is set to notmuch new – so when the IMAP run is complete, notmuch is called

notmuch new processes all the email it doesn’t know about, adding a set of tags to every new message

notmuch then checks for the file <maildir>/.notmuch/hooks/post-new and if it exists then it will run it

in my case, the hook file calls afew --new --tag

The tags added by notmuch new are set in the notmuch config file, in the part that looks like:

[new]
tags=new

So all new messages will have those tags added. Your post-new hook can then run only against mail that is tagged “new”, making it nice and fast. And as the final step, the post-new hooks should remove the “new” tag from all messages that have it, so that you don’t have to re-process them next time. A common idea is also to remove the “new” tag from messages that match filters you want to archive, and at the end of your filter list you replace the “new” tag with the “inbox” tag for any messages still tagged “new”.

You can do this with a basic shell script that calls the notmuch command line binary, as shown on the notmuch initial tagging page. However the command line binary only looks at a few headers (From, To, Cc, Subject, Date and message ID as far as I can tell). So I thought I’d look into:

afew

afew is a python command that will run filters against a set of messages and generate tags. I can install it using pip. However, at the time of writing the documentation is not as useful as it could be. I ended up reading the code to work out what was going on.

One key bit of information is that the config files are stored in ~/.config/afew/config which meant I could search github for path:afew/config – which only brought up two hits when I just tried it, but that’s a start to see what others do. The most useful one I found was by pazz and you might also want to look at my afew config.

When you’re playing with afew, running afew --help will get you started reasonably. And the command you want for the post-new hook is:

afew --tag --new

Which will run against the new messages and run the tag filters.

afew filters

I had to read the code to work out what some of the filters were, so here are my findings as a first stab at documenting them. Later I intend to improve the docs.

afew’s built in filters

The default filter set (if you don’t specify anything in the config) is:

These can be customised by specifying settings beneath them. The standard settings are:

message – text that will be displayed while running this filter if the verbosity is high enough.

query – the query to use against the messages, specified in standard notmuch format

tags – the tags to add to messages that match the query

tag_blacklist – if the message has one of these tags, don’t add tags to it

Note that most of the filters below set their own value for message, query and/or tags, and some ignore some of the above settings.

SpamFilter

This looks for the header X-Spam-Flag – if it finds it, the spam tag is set. You can override the tag used for spam if you want.

The settings you can use are:

spam_tag is the tag used to identify spam. It defaults to “spam”

ClassifyingFilter

This is to do with learning what tags match what without explicit rules. I haven’t worked this one out fully, but the project README has some details on how to use it. If I work out more I’ll blog about and link to that from here.

KillThreadsFilter

If the new message has been added to a thread that has already been tagged “killed” then add the “killed” tag to this message. This allows for ignoring all replies to a particular thread.

ListMailsFilter

This filter looks for the List-Id header, and if it finds it, adds the list name as a tag, together with the tag “lists”.

ArchiveSentMailsFilter

Basically does what says it on the tin. Though more accurately, it looks for emails that are from one of your addresses and not to any of your addresses. It then adds the sent tag and removes the inbox tag.

InboxFilter

This removes the new tags, and adds the “inbox” tag to any message that isn’t killed or spam.

FolderNameFilter

This looks at which folder each email is in and uses that name as a tag for the email. So if you have a procmail or seive set up that puts emails in folders for you, this might be useful.

Adding your own filters to afew

You can modify filters, and define your own versions of the base Filter that allow you to tag messages in a similar way to the notmuch tag command, using the settings above. Showing some sample configs is the easiest way to understand. The notmuch initial tagging page shows a sample config:

There were various python libraries and apps I wanted to use, and afew and alot in particular requires python 2.7

I am running all this on a machine that is running Debian Squeeze which ships with python 2.6 and a little research reveals that it would be unwise to grab python 2.7 from testing/Wheezy as it will then become the system python and break lots of things.

But then I came across pythonbrew which is inspired by the ruby rvm – it allows you to install different versions of python inside your home directory. However, doing this straight off might lead you to be missing some vital functionality related to compression or unicode (as I did the first couple of attempts). So the debian packages to install (covering all the packages I need for my notmuch journey) are:

At this point I was pretty much good to go, although I should also mention that you can save having to remember which version of python you are currently using by having wrapper scripts with contents like:

#!/bin/sh
$HOME/.pythonbrew/pythons/Python-2.7.3/bin/alot $@

Finally, I also came across a fork of pythonbrew called pythonz that I may look into further at some point. It support PyPy and Jython amongst others, but as it says in the README “[pythonbrew] has some extra features which I don’t really need, so I made this for to make something a bit simpler that works for me” so some things might be broken …

Password Storage

I’m running on a server and connecting over ssh, as I want the same set of tags to be available everywhere. I don’t want my password sitting on disk in clear text (particularly as it is my LDAP password, so used for sudo, ssh and various other logins). There are various solutions to this if you are running a desktop – python-keyring-lib will link to Gnome, KDE and OS X keyrings. But it took a bit more searching to work out how to do this on a server.

I considered a number of options

GPG encrypted file

I found this unix stackexchage post about having a python script that could decrypt a GPG encrypted file (one file per password). gpg-agent would run on the server so you didn’t have to enter your passphrase every time you need to run it. I got this running, generating a GPG key that I wasn’t actually going to use for email. That meant I was OK setting a long time out for gpg-agent – I set it to 24 hours by putting the following in ~/.gnupg/gpg-agent.conf

However I wanted to be able to leave offlineimap running and at some point gpg-agent would expire, so this wasn’t really what I wanted.

That said I did leave this system in place for msmtp, as that will only run when I send an email – so I will be logged in at that point and can enter my GPG passphrase into gpg-agent. The relevant line for the .msmtprc file is:

Leave offlineimap running in tmux

When you run offlineimap, if there is no password or passwordeval in the .offlineimaprc file, then offlineimap will ask for your password (provided the ui isn’t set to be Noninteractive). offlineimap can also be told to run repeatedly, without exiting, by putting autorefresh = 5 to re-run every 5 minutes. In between runs, offlineimap will just sit at the terminal waiting, but it remembers your password, keeping it in memory. Using this I could leave offlineimap running for months inside tmux on the server I use.

(Note that offlineimap will not ask for your password if the ui is set to be Noninteractive. And the autorefresh setting must be in the [Account X] section – I put it in the Repository section originally which didn’t work).

ssh keys and offlineimap preauthtunnel

Finally I worked out that I could talk directly to an imap command line utility on the remote server over ssh – using a passwordless ssh key to avoid any passwords. offlineimap is instructed to use it by the preauthtunnel option.

There are two main IMAP servers in the linux world, courier and dovecot. The relevant commands are:

There are two ways to do this so that you don’t have to enter the ssh key passphrase all the time. You could set up a password-less ssh key to do this, or you could leave an ssh session running so that offlineimap can multiplex its connections on to it.

A password-less ssh key leads to the possibility of abuse, but you can generate a new ssh key just for imap and specify it’s use by modifying the above command to look like:

Obviously this will only work for you if you have ssh access to your mail server, so it won’t be an option for all. But I’ve given some other options above, so I hope you can find something that works for you.

Other notes

sendmail over ssh

Another option for sending email to a remote server is to use the sendmail command over ssh (assuming you have ssh access to the server in question). First you need to generate a passwordless ssh key and copy it to the mail server:

In this case the command is the sendmail command for exim – I’m afraid you’ll have to work out what your own sendmail command is. Finally, in your mail client, tell it to use ssh as the sendmail command. The following works for both mutt and alot:

ssh -q -i /home/hamish/.ssh/id_mail_smtp smtpserver.example.org

Now whenever your mail client wants to send email, it will start up the ssh connection and print the email to it. This will be picked up by the sendmail command on the far end and off the email will go.

ssh ControlMaster

ControlMaster is an ssh option that means you can leave an ssh connection open with a master session, so that future ssh connections reuse the master session rather than having to start an entirely new connection. This should mean quicker send and receive. One way to specify it in your ~/.ssh/config file is:

This means that connections will be automatically started, will stick around for up to 4 hours, and the socket for your connection to mail.example.com on port 22 with username user would be at /home/username/.ssh/muxcontrol/name@mail.example.com:22 (don’t forget to create the directory). (And the default directory is the global /tmp directory).

However a gotcha here is that the connection to the host will use whatever ssh key was used for the initial connection, not the one specified at the command line. So if offlineimap runs first, using your ssh key tied to the imap command, then when you try to send an email using sendmail over ssh, then your ssh command will run imap rather than sendmail – and your email will go nowhere.

The solution is to specify a different ControlPath for these commands. So for offlineimap you could use:

This will keep more ssh connections active, but the right action will be done using your different ssh keys.

Checking IMAP mailboxes

I was moving folders around and was having trouble working out whether the folders were available but offlineimap was failing to find them, or if dovecot wasn’t showing me the folders. Eventually I found how to ask for a folder list over telnet. In short:

I recently decided to switch to using notmuch as a mail indexer. I’ve been a long time user of sup and love the command line interface, gmail-like threads and awesome search. However it can often be a bit cranky and slow, and it “doesn’t play well with others” – if the mbox files it uses are modified (say by a second imap client) then it needs to rebuild it’s index. I could live with all those and used it for my work email for about three years.

The real nail in the coffin is that it has mostly died as a project (though there have been a few new people stepping up recently). I have made quite a few commits to it over the years, and it gets the occasional bug fix, but the mailing list has dropped off to nothing over the years. There was some excitement when the original author of sup, William Morgan, announced heliotrope and turnsole as a client server version, but it never really gained traction.

So I decided to investigate notmuch – which is basically a C program and library which takes your email and gives it to Xapian (a full text search database – also used by sup under the hood) and then runs queries against it. But notmuch does … not much else, so I also need to fetch email, add the new emails to notmuch, run a mail client, send email and look up email addresses.

It’s taken a lot of research and playing with various tools. Some of those tools were great, but poorly documented, so part of the point of these blog posts is to give some info for other people to find. And I might even get round to adding to the documentation for some of these projects.

So here are the parts – I’ve put up a post about each major part, and all the small bits and links are part of this post.

two copies of notmuch

offlineimap and msmtp – including the thorny issue of password storage on a headless server (hint: use single purpose password-less ssh keys).

Two copies of notmuch

I run two copies of notmuch – one for my gmail account and one for my work one. I want to keep them separate, so I have set up a parallel config. The default file is ~/.notmuch-config and I have added ~/.notmuch-config-work You can tell notmuch to use it via an enviroment variable – $NOTMUCH_CONFIG – but I don’t want to set that all the time, so I have a file ~/bin/notmuchwork with the contents:

NOTMUCH_CONFIG=~/.notmuch-config-work notmuch $@

and then have an alias set up in my .bashrc:

alias nmw=$HOME/bin/notmuchwork

I have similar set up for alot and afew – although for those you can define the config path on the command line rather than through an environment variable. This is necessary as they access notmuch through the python library rather than through the command line.