you need to set the control_path option in your ansible.cfg file to tell SSH to use shorter path names for the control socket. You should have a look at the ssh_config(5) man page (under
ControlPath) for a list of possible substitutions.

In the company I work for we’re using RabbitMQ to offload non-timecritical processing of tasks. To be able to recover in case RabbitMQ goes down our queues are durable and all our messages are marked as persistent. We generally have a very low number of messages in flight at any moment in time. There’s just one queue with a decent amount of them: the “failed messages” dump.

The Problem

It so happens that after a botched update to the most recent version of RabbitMQ (3.5.3 at the time) our admins had to nuke the server and install it from scratch. They had made a backup of RabbitMQ’s Mnesia database and I was tasked to recover the messages from it.
This is the story of how I did it.

Since our RabbitMQ was configured to persist all the messages this should be generally possible. Surely I wouldn’t be the first one to attempt this. 😐

Looking through the Internet it seems there’s no way of ex/importing a node’s configuration if it’s not running. I couldn’t find any documentation on how to import a Mnesia backup into a new node or extract data from it into a usable form. 😞

The Idea

My idea was to setup a virtual machine (running DebianWheezy) with RabbitMQ and then to somehow make it read/recover and run the broken server’s database.

I edited
/etc/hosts to add $BROKEN_HOST to the list of names that resolve to 127.0.0.1.

Now restarting RabbitMQ failed with yet another error:

1

2

3

4

5

6

7

8

9

10

Error description:

{could_not_start,rabbit,

{{schema_integrity_check_failed,

[{table_attributes_mismatch,rabbit_queue,

[name,durable,auto_delete,exclusive_owner,arguments,pid,

slave_pids,sync_slave_pids,recoverable_slaves,policy,

gm_pids,decorators,state],

[name,durable,auto_delete,exclusive_owner,arguments,pid,

slave_pids,sync_slave_pids,mirror_nodes,policy]}]},

{rabbit,start,[normal,[]]}}}

Now what? Why don’t I try to give it the Mnesia files piece by piece again?

Reset $RABBITMQ_MNESIA_DIR

Stop RabbitMQ

Copy
rabbit_* files in again and fix their permissions

Start RabbitMQ

All our queues were back and all their configuration seemed OK as well. But we still didn’t have our messages back yet.

The queues have been restored, but they have no messages in them

Solution

So I tried to copy more and more files over from the backup repeating the above steps. I finally reached my goal after copying
rabbit_* ,
msg_store_* ,
queues and
recovery.dets. Fixing their permissions and starting RabbitMQ it had all the queues restored with all the messages in them. 😂

Queues and messages restored

Now I could use ordinary methods to extract all the messages. Dumping all the messages and examining them they looked OK. Publishing the recovered messages to the new server I was pretty euphoric. 😁

If you use Python‘s Bottle micro-framework there’ll be a time where you’ll want to add custom plugins. To get a better feeling on what code gets executed when, I created a minimal Bottle app with a test plugin that logs what code gets executed. I uesed it to test both global and route-specific plugins.

When Python loads the module you’ll see that the plugins’
__init__() and
setup() methods will be called immediately when they are installed on the app or applied to the route. This happens in the order they appear in the code. Then the app is started.

The first time a route is called Bottle executes the plugins’
apply() methods. This happens in “reversed order” of installation (which makes sense for a nested callback chain). This means first the route-specific plugins get applied then the global ones. Their result is cached, i.e. only the inner/wrapped function is executed from here on out.

Then for every request the
apply() method’s inner function is executed. This happens in the “original” order again.

Below you can see the code and example logs for two requests. You can also clone the Gist and do your own experiments.

If you find yourself–like me–in the situation that your Mac has crashed and you want to retrieve the crash reports (some call them logs 😉 )? Well, there are basically two ways.

You can look them up with the “Console” tool (find it in
/Applications/Utilities/Console or with Spotlight). Open the “System Diagnostic Reports” section on the left and find an entry similar to
Kernel_<date>_<your_pc_name>.panic at the top.

You can also find these reports as text files under
/Library/Logs/DiagnosticReports with the same names. OS X will open them with the Console tool per default.

In my work – every now and then – I found myself in need of a browser with reduced security checks (mainly to gloss over cross domain XMLHttpRequests and SSL certificate violations) for testing purposes. I didn’t want to take the risk and use my main browser session with these settings, so I made me a script (also available as a Gist). 🙂

Sometimes well-intentioned features have unintended side effects. Case in point: WordPress’ maintenance mode. Whenever you update plugins WP will automatically enter maintenance mode, which displays a nice message to your visitors that the site will be back online shortly. It will automatically go out of maintenance once the updates are done.

Well, sometimes unexpected things happen: you are stuck in maintenance mode. WP will effectively lock you out … even the admin section will not be accessible. *ugh* This is the moment you start panicking … luckily if you wait 10 minutes or delete the .maintenance file manually you’ll be able to access your site again. *phew*