Paperless Appreciation

My Paperless project has been running for years now, and every once in a while someone discovers it and posts a link to Reddit or Hacker News and there's a sudden uptick in people trying out the software. When interest spikes like this, every once in a while I get a lovely email like this one and it makes my day.

I'm posting it here as a reminder to myself and others that while our labours on Free software projects may sometimes feel thankless, sometimes they help.

Hey Daniel,

I'd like to reach out and thank you for writing paperless.

While I like having things stored nicely and in a way that makes it easy for me to find them again, I have to admit that I'm not really a talent in doing so. Especially when it comes to paper.

Now I'm running paperless on an Odroid XU4Q and I actually manage to archive documents within the same month I receive them instead of moving them from pile to pile for a year or two and then spend whole weekends on filing them in folders.

In case you happen to visit Zurich one day and need information or help of any kind, please let me know. I'll be more than happy to help :)

Thanks again
Cheers

p.s. I have donated CHF 100.- to UNHCR as a "Thank you".

It was the last line that really did it for me. I added a call for donations to the UNHCR to the bottom of the project's README.md file back in 2016 and pretty much forgot about it afterward. Now, more than a year later, someone helped refugees thanks in part to code I wrote.

The sudo:immerse Hackathon

Months ago, I signed up for a local hackathon
and then promptly forgot about it. Then, the night before I was due to attend,
I realised that it was actually a VR hack, and considered not going. I'm a
web guy after all, and didn't fancy the idea of using Windows to play with
Unity for two days.

Christina convinced me to to go, and I'm so glad she did.

It was a great event. We took over the RealVNC
offices for 48hours to first learn about the capabilities of emerging VR
hardware, and then to attempt to build something useful with it. I played with
an HTC Vive and
Microsoft Hololens and was
introduced to a woman from Give Vision who
inspired some of us to develop a project for the visually impaired.

I fell into a team of brilliant, committed people and early in the process we
realised that we all more-or-less shared the same idea for the hack:

We were going to build a tool to first help diagnose people with macular degeneration,
then build another tool to help those suffering from it cope with the
disease.

Our team was comprised of six people, all of whom just showed up for the event
looking for something to do. In fact, nearly every other person at the evnet
arrived with a pre-defined team and product in mind. We were effectively the
leftovers.

We broke into 3 sub-teams, with two people each:

Luke & George worked together to build the test front-end and coping tool
respectively

Peter & Clare researched the industry, developed use-cases, and prepared a
presentation

Ullash and I wrote the mapping system for the test as well as the algorithm
to zero-in on a user's blind spot at higher and higher resolutions.

By 11pm Saturday night, each team had more-or-less completed their side, and by
Sunday at 3pm we were ready to present.

...and we won.

The finished product is effectively two separate products focused on helping
people with macular degeneration:

The test is a simple web app (written in Javascript) that runs on any phone.
It's designed so that you can take your phone, plug it into any Google Cardboard
device, and like magic you have an eye test that maps your blind spot(s) and
will even email that map to your optometrist.

The visual aid works the same way, via Google Cardboard, but takes your blind
spot map as input. It taps into your phone's camera to give you a real-time
view of the world, literally bending the image around your blind spots to
help you see.

The finished product(s) has all manner of benefits:

Reduce the costs of medical care by reducing routine visits for testing

Improving the mobility and independence of those suffering from the disease.

Increase the amount of data collected on this disease through the historical
charting of the disease's progression.

Increase understanding, by allowing others to see what their relatives see
using our app.

I'm really proud of the team. We built a product that's not only useful, but
accessible. The total cost of this thing is that of a smart phone + $5 for
Google Cardboard. This could be deployed around the world to help detect signs
of macular degeneration literally years early allowing treatment before it
progresses too far. It'll help parents stricken with the disease keep tabs on
potential signs in their children (this is a genetic disorder) and all this
done with a phone and the price of a Starbucks coffee.

Our prize was a Google Home, one for each of us. Honestly though, I don't
think any of us much cared about the prize at the end of the day. We were
exhausted and thrilled at what we were able to build in such a short amount of
time.

The finished product is in a state one might expect from a hackathon: patched &
working, but in no way ready for public use. The code is on Github
and there's a few live samples if you wanna give it a try:

Moving car: Tap the
screen to see what the world is like for someone with MD. The panel on the
left is what they normally see. The one on the right is with our video
distortion.

Live stream: You
have to enable your camera permissions for this one, but this will
demonstrate how the app works in real time. Tap the screen to switch though
the few modes we setup.

The test: The
actual test. Dots appear at random locations and at random times. Tap the
screen to record the fact that you saw it. Hits & misses are logged in the
app and mapped internally. PDF generation works, but it's flaky.

What happens next is still unclear. The product as-is isn't ready for the
public, but no one person on the team is really capable of picking up the whole
thing and running with it. Give Vision might
take it over, or maybe we'll all get together in a few weeks and polish it up a
bit. I don't know.

The organisers said they'd be doing another event like this in March next year
though. I hope to attend that one as well.

I (We) Got Married

I probably should have said something back in August when it happened, but we
had that was a chaotic month or so, and then we were waiting on the
photographer to send us her stuff, so now, over 2months after the fact, here's
the official post.

On August 25th of this year Christina & I were finally married. The ceremony
was a simple civil one, and we had about 25 guests representing no fewer than
19 different nationalities. We had friends there new & old, and our immediate
families both made the trip from their respective homes of Athens & Kelowna.

The Big Day was lovely, with excellent weather and zero problems from any of
the various moving parts required for such an event. Christina's stylist did
a wonderful job, the people running the actual ceremony had everyone in & out
of there like a well-oiled machine, and our photographer was a pro through the
whole thing, setting us up for shot after shot, getting lots of candid photos
and just generally being awesome.

In terms of cost, everything was surprisingly reasonable with everything: the
photographer, stylist, bouquet, ceremony, rings, dinner & venue coming in under
£4000. For a wedding, I understand that that's pretty amazing.

I feel like this post should be a bigger deal, but it just doesn't feel like
that. In retrospect, our wedding -- our marriage, feels like an inevitability,
and the whole ceremony more of an excuse to have a party than anything
monumental. Of course rationally, this is most definitely a Big Deal --
there's a reason LGBT activists continue to fight for marriage equality around
the world, but for me the legal stuff has never really felt consequential.
What matters is that I was committing to her, and I'd already done that years
ago, you know?

After the Big Day, we all relaxed for a few days and then hopped on an abysmal
Ryan Air flight to Athens where we were met at the airport by Christina's
extended family. The swept us off to her aunt & uncle's home where we were
treated to a fantastic traditional Greek feast. My parents, heads still
spinning from landing in a foreign country fell in love with the food
immediately. Christina's aunt is fricking genius in the kitchen and she had
pulled out all the stops.

We then had a week of sightseeing in Athens, followed by Wedding #2 in an
outdoor restaurant alongside Lake Marathon. Most of the guests were from
Christina's side of the family, but a few of our friends from Amsterdam also
made the trip down to celebrate with us. The next day, Christina & I toured
Athens with the Amsterdammers before they all had to go home.

At that point, we all split up: my parents went on a cruise to a number of
Greek islands, Christina and I went on our honeymoon to Ios, and Christina's
parents took a week off from the craziness to relax a bit.

Ios is beautiful and much less hectic than other, more touristy islands. The
food is wonderful, and the sightseeing was lovely. For my part though, all I
wanted to do was relax by the pool and I managed to do lots of that :-)

After a week of that, we all returned to Athens for another week of
sightseeing, and then finally returned to Cambridge where my parents celebrated
their own wedding anniversary with their son & new daughter in-law.

It was just wonderful that so many of you were able to make it out to celebrate
with us. Honestly, that's going to be the most memorable part of all of this
for me. As an expat, you rarely have the opportunity to reconnect with all the
people you love, because you're all scattered around the globe. It's one of
the things that make weddings awesome.

There will be a Wedding #3 though: this one in Vancouver next year, likely in
July (we're still working on exact timing). If you couldn't make it to #1 or #2,
come on down to YVR for #3! We'll be doing what I'd always said I wanted for
my wedding: a potlach in the park. There will (hopefully) be sunshine, and
KFC, and my dear old friend Michelle has promised to sing with me for the
occasion. It should be a good time. I'll post more when I know more.

For now though, check out the photos! These
are the ones the pro took on the Big Day and these
are the ones the rest of us took throughout the multi-week event.

Public Key Authentication for Media Files: Why Isn't This a Thing?

I'm just writing down my thoughts here in the hopes that Someone Smarter Than Me might be able to shed some light on the idea, or perhaps even work with me to make it happen.

I'm reading more and more about how fake news stories are circulating, and how technology has developed to the point where we can literally create images, audio, and video of events that never happened but appear as though they did. The effort so far seems to be in the area of somehow detecting a fake by searching for evidence of tampering, but this to me feels wrong-headed: it's expensive, slow, and will always be a step behind the fakes.

Why instead do we not simply sign each file on a sub-channel so it can be easily proven to be legit from the source?

For example, the BBC does a story about a politician and includes with it a picture of her doing something interesting. This picture is then circulated around the web with two bits of information hidden inside the EXIF data:

The original source organisation (BBC)

The signature of the image based on the BBC's private key

The original URL of the image (maybe?)

The image is then re-shared onto Facebook, where they've got simple software that:

Reads the original file and authenticates its origin against the BBC's public key

Resizes the image for its own purposes

Appends a second signature using Facebook's private key

Posts the video into the user's timeline with a "Verified BBC image, resized original from Facebook" caption

If the image is re-shared onto Twitter, or Google+, or Diaspora, these services will only be able to know that the image came from Facebook, but theoretically this still means more than not knowing the origin at all.

The goal is to create a means of authenticating the original source -- or at least a source more credible than "Jim's computer", and perhaps even the chain of modifications to said source There's also no reason this couldn't be applied to all kinds of media.

Maybe this technology already exists, though a cursory search didn't turn up anything for me. Anyone have any bright ideas?

django-debreach + DRF = sadness

I sunk 4 hours of my life into this problem yesterday so I thought I might post it here for future frustrated nerds like myself.

If you're using django-debreach and Django REST Framework, you're going to run into all kinds of headaches regarding CSRF. DRF will complain with CSRF Failed: CSRF token missing or incorrect. and if you're like me, you'll be pretty confused since I knew there was nothing wrong with the request. My token was being sent, but it appeared longer than it should be.

So here's what was happening and how I fixed it. Hopefully it'll be useful to others.

Django-debreach encrypts the csrf token, which is normally just fine because it does so as part of the chain of middleware layers in every request. However, DRF doesn't respect the csrf portion of that chain. Instead it sets csrf_exempt() on all of its views and then relies on SessionAuthentication to explicitly call CSRFCheck().process_view(). Normally this is ok, but with a not-yet-decrypted csrf token, this process will always fail.

So to fix it all, I had to implement my own authentication class and use that in all of my views. Basically all this does is override SessionAuthentication's enforce_csrf() to first decrypt the token:

DjangoCon 2017

I love DjangoCon. I've been going to it almost every year since I arrived in
Europe back in 2010. Sure, a considerable portion of my career has been based
on Django, but it's more than that: the community is stuffed full of amazing
people who genuinely want us all to succeed and that just makes the conference
all the more exciting.

This year we all converged on Florence for three days of talks in a historic old theatre
at the heart of the city and like every year, the talks at this single-track
event were hit-and-miss -- but that's ok! When the talks were less-than-useful
we could always just pop out for gelato or catch up in the hallways with other
developers.

The Good

Community

From talks covering gender bias or autism, to the re-labelling of all bathrooms
to be unisex, DjangoCon has long been a shining example of how to be inclusive
in a software development community and it's something I'm proud to be a part
of. This year, they even raised enough money to pay for flights and
accommodation for a number of people from Zimbabwe who are trying to grow a
local Django community.

It feels good to be part of a group that's so welcoming, and I would argue that
IT, while traditionally straight-white-male-dominated, is uniquely suited
for the multicultural mantle of tolerance. Every other field has a uniform:
a standard by which you're judged as "in" or "out" (just watch London's
financial sector at lunch hour they all wear the same thing). In the
software world however, we're all defined as being the odd ones. We are the
all-singing, all-dancing nerds of the world: our differences are what make us
fabulous. DjangoCon embraces that in a way I've not seen anywhere else and I
love it.

Talks

Level up! Rethinking the Web API framework: Tom Christie

Tom Christie is the genius who brought us Django REST Framework and he's now
working to improve the whole process by taking advantage of Python 3's type
annotations to make your code self-documenting and then use that
self-documentation to better build a browseable API. His code samples were
beautifully simple and I'm very excited about the future of DRF. He's doing
some great work there.

The Art of Interacting with an Autistic Software Developer: Sara Peeters

This was one of those talks that really felt as though it was lifting
metaphorical scales from my eyes. Like many software engineers, Peeters is
autistic, but unlike too many such people, she's extremely self-aware and
articulate about what this means for her own human interactions.

She walked us through an average day for her: how she chooses her route home
not based on the efficiency of the route, but because it limits the intensity
of crowds on her commute as well as the chance that she'll encounter rain.
It's the sensory overload you see, the idea of so many raindrops impacting her
skin like that is a terrible feeling.

In 20min she helped paint a picture of the limitations and fascinations of
dealing with autism in her day-to-day life, and outlined a few ways the rest of
us might help communicate and accommodate people in her situation.

After her talk, I found myself thinking back on a few former coworkers.
Perhaps if I'd been more understanding, and if they'd been self-aware enough
to help me understand their needs, we might have gotten on better.

The OpenHolter Project: Roberto Rosario

This talk blew my frickin' mind.

The guy has a severe heart condition which left him bedridden for 23hours a
day, and he's managed to make his life liveable with $30 worth of equipment and
some Free software.

His talk walked us through the process of building your own mobile EKG machine.
A device that normally costs thousands of dollars and typically only used in
a hospital, Rosario built with an Arduino and parts
he bought off the internet.

He then showed all of this to his doctor who asked if he could develop a diary:
basically a log of his heart rate throughout the day, annotated with
explanations as to what he was doing when anomalies appeared in the log.

He managed this by having his little device push daily log data onto his Django
stack where it was all neatly logged and charted:

That's 100 samples per minute of biometric data generated by yourself on a desk
in your house for $30 plus the cost of cables. This future we're living in
is amazing.

Autopsy of a Slow Train Wreck: Russell Keith-Magee

Russell ran a start up from optimistic start to a brutal, crushing finish years
later, and decided to do a talk to teach us all what went wrong.

The talk was broken down into succinct sections, with a lesson in each case. A
valuable talk for anyone considering a future in a small business. When it's
made available online, I'll be sending it around to a few people I know.

Fighting the Controls: Daniele Procida

Daniele wrapped up the event with a final talk about a plane crash, or maybe it
was Icarus -- it's hard to explain. His message was simple though: bad things
happen when you don't stop and consider what's happening.

When stuff is exploding, the server is on fire, and everything is falling
apart, sometimes the best thing to do is to just sit there and breathe:
consider the situation and act when you have a better handle on things.

His talks are always a delight, as he has a unique way of humanising
software. Once the videos are live, I recommend this one to anyone in any sort
of high-stress job.

The People

Meeting the developer of Mayan EDMS

About a year ago now, I was sitting in a London pub, hacking away at my latest
project, Paperless when I stumbled
onto Mayan EDMS: another open source project that
did almost exactly the same thing as mine, but it was prettier and more
featureful.

I was crushed. Here I was pouring literally hundreds of hours into this thing,
with thousands of people using the code through GitHub, and suddenly, it all
felt like it was for nothing because someone else had done it all already.

The guy who wrote that thing? I met him over lunch on the 2nd day of
DjangoCon. He's also the same genius who built the mobile EKG machine
mentioned above.

It was fun to meet him, talk about what worked for him and what didn't, and
what sort of future he has planned for Mayan. He's a pretty smart dude, and it
was nice to just sit and chat with a sort of "rival" nerd.

Talking to Paperless contributors

I also ended up talking to Philippe Wagner,
one of the Paperless users who's been quite helpful in pushing the project
forward. He wants to repurpose Paperless into a sort of markdown-based
Evernote clone, and to do that all he needs from me are some minor changes to
the project core to make it more pluggable. We'd been talking about it in the
GitHub issues queue for a few weeks and he recognised me in the DjangoCon Slack
channel, so he sent me a private message asking if we could chat for a bit.

I stepped out of one of the less interesting talks and we worked out a plan to
make things work just outside the theatre. He's a cool guy and very driven.
It's great to have him working on Paperless.

New Friends

After the first lunch, I sort of fell in with a group of fun people for the
rest of the conference. We hung out after hours looking for food or just
company for a walk around town. This is uncommon for me as while I'm a
relatively friendly person, I generally avoid people save for superficial
conversation. This was a nice change.

The Bad

Questions

The event was really squeezed for time and almost every talk didn't allow for
questions. Instead, we were directed to the Slack channel (which was only good
for people with working wifi and laptops for fast-typing) or "later around the
conference". Personally, I've always liked the questions, as it allows the
audience to get the speaker to publicly defend an assertion or elaborate on
something. Without it, it felt really disconnecting, as if I just watched the
talk on YouTube.

Language

While I think that DjangoCon should be celebrated for its adoption of a code
of conduct and for its inclusive attitude, I feel that it's fallen into that
ugly trap of adopting a language police. In an effort to be an inclusive
community, they're effectively rewriting the dictionary.

Specifically, I'm most annoyed by the policing of the word "guys" in
reference to a group of people regardless of gender. I get that our community
is composed of men and women, and people who defy gender labels, but I don't believe
that that means that we need to strip non-aggressive language to accommodate
some people.

In the same way that we don't censure people for talking about hamburgers
around vegans, your comfort with my words is not my problem. Of course this
isn't a defence of racial slurs, aggressive language, threats or hate speech --
that's totally inappropriate for an open and tolerant community, but I think
that this business of reducing language based on the comfort of a few is a
threat to the free exchange of ideas, not to mention entirely tone deaf to the
fact that at least 70% of the attendees to DjangoCon were non-native English
speakers who rightly use this word in reference to any group of people
regardless of their position on the gender spectrum.

The worst part of all of this is that by simply discussing my distaste for this
practise, especially at the conference, I risk being ejected from the community
like some sort of nerd heretic. I maintain that it's dangerous and unhealthy,
but I had to wait until now to say anything because I didn't want to be kicked
out of the event. This can't be conducive to a Free and Open society, let
alone a conference.

Conclusion

So to wrap up: some good, some bad, but on the whole, I'd say it's was well
into the good column. I'll be back next year, and maybe I'll even try to give
a talk on something.

2016

This was a big year, bigger than I had remembered when I sat down to write this
thing. Somehow, I'd forgotten about half of this stuff, and rolled the other
half into 2015 in my head. But 2016 wasn't all terrible. Here are the
highlights:

Personal

2016 was a big deal on the where-to-live front. I finally got my wish and we
moved away from the Netherlands and into a real city: London: The Centre of
the World.

It turns out however that London is a rat infested toilet drowning in social
inequality in a country rife with xenophobia, nationalism, and a dangerous mix
of pride and ignorance.

Yes, you can quote me on that.

Our flat was a dark, damp, rat infested hellhole with a ground-floor view of
a wall (the British love walls and fences almost as much as they love
classism). The Tube is a remarkable feat of marketing that has managed to
brand a hobbit tunnel of loud, stinky, smoggy,
dampness as "modern" and "cultured". And absolutely everywhere you go, there
are homeless people, stepped over and ignored: immediately by the public, and
systemically by the government. They even have a quaint British term for them
so it doesn't sound so tragic: rough sleepers.

London is amazing if you're a tourist, but once you live in its decaying
buildings, commute on its antiquated transport, or are forced to breathe its
toxic air for more than a few days, you recognise it for what it is: a terrible
place to live.

...which is why we moved to Cambridge

The air is cleaner here, the roads more bike-friendly (though it has a long way
to go before achieving Amsterdam-level cycling support) and nearly everything
is walkable. It's a town more-or-less built for people, as opposed to London,
which is built for plebs.

Our flat here is in a lovely modern building with proper ventilation and
underfloor-heating. It's cool & quiet in the summer, warm & dry in the winter,
and my commute is 30 minutes by bike along the river. Christina rides her bike
through town in about 12 minutes.

Come visit us in Cambridge. You'll wonder what the hell everyone is doing in
London.

Travel

As with every year I've lived in Europe, I did a reasonable amount of
travelling this year, and once again, it feels as though I didn't travel
enough.

Amsterdam

We may have left town, but Christina still had to return to defend her PhD in a
process that's part ceremonial (you should see the Wizagamot-esque robes) and
part academic (she literally had to defend her PhD against questions from
academic rivals and friends). Unsurprisingly, she dominated the event, and
walked out with a shiny new piece of paper attesting how brilliant she is.

Αθήνα & Μετέωρα

Right after Amsterdam, we hopped on a plane to Greece for Orthodox Easter where
I once again ate far too much food and enjoyed the sunshine. This holiday
included a road trip out to Μετέωρα
where we did a little hiking and sightseeing around the monasteries in the
area.

Copenhagen

The bi-annual RIPE Meeting was held in Copenhagen and as they had a hackathon
for monitoring software, I signed up to play -- and my team won!
Our project was called HALO,
a heads-up display for your network, and the source code is here
if you're curious.

Sesimbra

Christina's friend Ana got married in Sesimbra, Portugal this year and I'm so
glad that I was able to attend. The wedding was lovely, and the country,
beautiful. The food was good, the people friendly, and the view from our hotel
room was awesome. Twitter has a few pictures.

Vancouver & Kelowna

The biggest news of the year is of course that my niece, Lucy was born! I was
careful to time our trip home to coincide for her birth, but she had the
indecency to be born a couple weeks premature, so when we finally showed up, we
got to visit her in the hospital.

The trip home was also an opportunity to introduce Christina to Vancouver in
the summer time. We also had an engagement party there so my family that can't
make the trip to Greece would have an opportunity to spend some time with
Christina. There's a great big blog post about it
if you're curious.

Brussels

I was in Brussels twice in 2016. Once for my annual trip to FOSDEM,
and later for Freedom not Fear, a series of
meetings & workshops around freedom, surveillance, and politics in the EU. The
former was great (it always is), and the latter, combined with my experience at
Mozfest this year has given me some serious
insight into the nature of EU politics. I want to do a separate post about
that later though.

Madrid

There was another RIPE Meeting in 2016, and I showed up for that hackathon too.
We didn't win though -- I think -- I had to leave before the announcements, but
I don't think we did. The project was called "Pinder" or, Tinder for
peering and the presentation is here,
the code here, and an explanation over
at peer.sexy.

Αθήνα, Again

One last trip back to Greece this year to make up for all the time Christina
lost while working on her PhD. This was primarily a Christmas trip, so it was
all just meeting with family, eating far too much, and exchanging presents. I
also used some of the down time to work on my own family project that I
mentioned in a previous post: my grandpa's video archive.
There are some photos here
if you're curious.

Professional

This was a big year for me professionally. I started contracting, started
working for government, and took on a lead role at another company. I also
almost got a job I desperately wanted, so I'm including that here too.

UKTI

The move to the UK started with my first (and likely last) government job ever.
This was big money and a big title combined with everything you've heard about
government work and more. I have never been more angry and frustrated on the
job than I was there, but I probably would have stuck it out were it not for
the fact that they were selling weapons to people what shouldn't have them.

Gynii-Me

In parallel to my work at UKTI, I started helping out a brand-new start up with
occasional technical advice in what their options were for building a women's
fertility web platform. I don't get paid, but I do help out where I can,
vetting agency proposals and explaining complex technical topics to the company
CEO. It's a fun side gig, and they're good people so I'm happy to help where I
can.

Cyan

I moved from UKTI to a company called Cyan/Connode who were super-convenient,
as they had a London and Cambridge office and we were moving up there in a few
months. I helped them out on the technical front, and helped management
understand a little about why they were having retention problems, but was
terribly unhappy, so I got out of there after a few months.

Mozilla

In my quest to get out of Cyan, I applied to Mozilla for what would have been a
pretty amazing position: engineer on the incredibly popular Mozilla Developer Network.
Unfortunately, while I made it to the very last round, I didn't get the job,
which sucked, but it was an honour to make it that far anyway.

Money Mover

I ended up moving on to a fintech company that has an office just outside of
Cambridge and wonderful staff of truly friendly and engaged people. Seriously,
best work environment ever. It's a small team right now, but we'll be
growing in 2017. My role is Lead Developer which is pretty fabulous, and my
current Big Job is picking up the code left from an agency that did the bulk of
the work for the company over the last few years and making it ours. Having
worked at a few agencies in the past, I suppose I deserve this :-/

Projects

Like every year, I overextended myself on New Projects as well as building on
the old.

Paperless

Early in the year, I suddenly lost interest in my super-popular project, Paperless
when I discovered that there was an eerily-similar project
out there doing things better than I had. I didn't really do much more than
field pull requests for much of the year, but toward the end, there seems to be
a lot more interest all of a sudden, and I've started doing a little more work
on it.

There seems to be a "market" for a project like Paperless which is much less
complicated and capable of running on lighter hardware.

Korra

Working for government introduced me to the clusterfuck that is "security" in
large office environments. I wrote something fun & easy to self-host and it
got a reasonable amount of attention on Reddit and at the London Django meetup.

Basically, Korra lets you share files
easily, without special software, and securely so that you don't have to do
insane things like email people's passports or private government documents
around.

Spirithunter

When I started commuting longer distances (to Cambridge from London for while)
I started back in on Spirithunter,
trying out Django's new Channels support (OMG it's awesome). However, when my
commute shrunk to a 30min bike ride, all of that development stopped. I might
pick it up again when I'm bored one day, or if Mihnea decides he wants to hack
on it with me.

Global

I know that this is a personal blog, so it seems kind of silly to reflect on
global events here, but these things affect me, so I thought it relevant.

#Brexit

What a disaster. After living here only a year, I'm not surprised at all that
this country voted to Leave the EU, but I'm still saddened by it. It will take
generations for this country to recover from this mistake, and knowing what I
now know about British culture, I'm sure they'll find a way to look back on all
of this as some sort of Trial they all had to go through, that they survived
because Britain Prevails or some doublespeaking-fluff like that.

I'm more concerned about the rise in hate-crime here though, and the remarkable
tendency people here have to blame immigrants for everything wrong with the
country -- especially when it's plainly obvious that the current government's
malevolent domestic policy is really what's at fault.

Trump

I called it and now it's going to
happen. As an outsider, I kind of want to sit down and watch everything burn with
a bowl of popcorn; this is after all what the public voted for. He, along with
the Republican House and Senate are going to hollow out the US and give people
everything they asked for. I can only hope that they serve as a cautionary
tale for the rest of us.

Deaths

A lot of important people died this year: Bowie, Prince, and Castro to name a
few. For me though, this will always be the year Leonard Cohen
died. The world is diminished without him in it.

Of course Rob Ford and Antonin Scalia died this year too. I'm really not all
that bothered by that. I suppose that's one of the greatest things about the
Reaper: he doesn't care who you are. When it's your time, that's it.

Conclusion

So that was 2016. Hopefully at this time next year I'll be posting about how
in 2017 I finally got Romanian citizenship, and how Christina and I finally
have a date & location for our wedding.

I'd like to do some more travelling to undiscovered (by me) places this year.
At the very least, I'd like to see more of Scotland and maybe even Romania and
the Czech Republic. None of that is booked yet though.

Grandpa's Archive

I built a thing for my family this Christmas and I wanted to post about it
briefly.

If you're one of the few people dedicated enough to follow this blog, you'll
know that my grandfather died last year,
and that he was sort of the family videographer.
What you likely don't know however is that this year, on my trip home
I acquired his entire collection of DVDs that he'd been accumulating over the
years.

This some really old stuff:

Around the Christmas tree when I was 3 or 4 years old

My dog learning tricks for the first time

My parent's wedding

My graduation

My mother as a child in Romania

My grandparents, so much younger, with friends in Romania

My niece, Violet

It was an amazing collection spanning 4 generations over 39 DVDs, and I spent
a few days on that trip home ripping every last one of the disks onto a
portable hard drive so I could take the raw data home for a special project.

Well that project is now finished, so for those of you who don't care about the
technical aspects, here's the link. I
shared the URL with my family by email on Christmas day since I was on the
other side of the world for the holiday festivities this year, but all in all,
it seems to have gone over well.

My father has suggested that I expand on the collection with my own videos in
the future -- I may just do that, though I'm more of a still photos guy. We'll
see.

The Technical

This whole thing was a HUGE pain in the ass, so I want to document the process,
perhaps if only for future websurfers looking to do something similar.

The Problem

The videos were in DVD format. Thankfully, it was digital, but it's certainly
not web-friendly. The video data needed to be ripped from the disks and
compressed into a web-friendly format that was high-quality enough to preserve
the video, but in a file small enough to stream to Canada-quality internet
connections.

Also, the DVDs were terribly organised and not indexed in any way. The disks
often had multiple title tracks, sometimes duplicate tracks, and there were
tracks that just contained garbage data.

Oh, and there was a time constraint. I only had the disks for a few days when
I was in Canada. I wasn't going to take them back to the UK with me.

The Process

It was basically done in three stages:

Raw DVD > .iso file > .webm file

The .iso file step was just a clean & easy way to back up all of the DVDs
without having to worry about accidentally missing something while I was
hurriedly trying to get through them all in Canada. By turning 39 DVDs into
39 files on a USB drive, I could be sure that I wouldn't accidentally lose
data during the ripping process.

As it turns out, this was a good plan, since it took a few weeks of tinkering
with this project before I realised that some disks had multiple titles on
them.

The creation of the .iso files was easy. I just put the disk in the USB DVD
drive I brought with me and typed this:

$ dd if=/dev/dvd of=/path/to/usb/hard-drive/disk-00.iso

Waited about 20min, then took the disk out, and repeated this... 39 times.

The creation of the actual video file on the other hand was the big problem.
There are lots of sites out there that claim to tell you how to do this, and
very few of them have anything helpful. I think that this is because the end
goal is rarely understood up front. Sometimes people are trying to encode DVDs
into a high quality file for local playback, and the settings for that are
rather different from what someone would want to do to encode for a
web-friendly format.

There's also a wide variety of tools out there, most of which are buggy,
unsupported, don't have a port for Gentoo, or just plain suck. The most common
recommendation I found was for Handbrake, which is an
impressive GUI for ripping videos but for me:

It didn't encode files that were high enough quality given the file size

It didn't make web-friendly formats. Even when you tick the box to make it
web-friendly, the output file doesn't stream in Firefox. I didn't test
other browsers.

It was terribly slow to find all the tracks, apply the settings I wanted and
then wait to see if things panned out. There's no command-line interface to
make things easier.

All of this lead to a lot of frustration and weeks of tinkering, finally
leading me to a site that gave me the magic ffmpeg incantation to generate a
web-friendly file:

Of course this assumed a .mp4 input file, and I wanted to rip straight from
the .iso, so after much digging, I discovered that ffmpeg has a means of
concatenating (chaining) video inputsandit can read straight from a
DVD's .VOB file. With this nugget of knowledge, all I had to do was mount
the .iso locally and compile a list of files conforming to this naming
convention:

VIDEO_TS/VTS_01_#.VOB

With that information, I wrote a quick shell script that ended up generating a
great big queue file
of commands that look a lot like this:

Unfortunately, ffmpeg doesn't really do threading very well, and the prevailing
advice out there appears to be that you should just thread the process yourself
rather than ask ffmpeg to try to use all your CPUs itself. For this bit, I
wrote a very simple paralleliser in Python
and magically, all of the cores on my super machine could crunch Grandpa's
videos, 16 at a time.

Finally, I wrapped the whole thing in a simple script
that mounted all of the .isos simultaneously and then ran the paralleliser,
and ran that in a tmux session so I could get on a plane and Fly to Greece
while my computer did its thing for two days.

While I was in Athens, I spent a day or two fiddling with the site itself,
getting video.js to work the way I wanted it to and
playing with Select2 to try and get an interface
that the non-technical people in my family could follow. I wish I had better
skills in this area 'cause frankly, the site is kinda ugly, but at least it's
functional now.

So that's it. I hope that one day, someone will find this stuff useful. The
ffmpeg incantations were especially difficult to find and assemble, so I figure
that'll help someone eventually.

neveragain.tech

Something amazing is happening in my industry right now and I want to
take a minute to talk about it.

Americans are freaking out. They're staring down the barrel of the very
real possibility that the Trump administration will draw up lists of
Muslims living the United States. This is a dangerous first step toward
dictatorship and the end of rule of law, and those of us paying
attention are understandably worried.

The American engineering community is slowly arriving at a state of self
consciousness though: a few of them have banded together and written
a pledge stating in short, that if the US
government wants to build technology to destroy the country, then
they're going to have to find someone else to do it:

We, the undersigned, are employees of tech organizations and companies
based in the United States. We are engineers, designers, business
executives, and others whose jobs include managing or processing data
about people. We are choosing to stand in solidarity with Muslim
Americans, immigrants, and all people whose lives and livelihoods are
threatened by the incoming administration’s proposed data collection
policies. We refuse to build a database of people based on their
Constitutionally-protected religious beliefs. We refuse to facilitate
mass deportations of people the government believes to be undesirable.

We have educated ourselves on the history of threats like these, and
on the roles that technology and technologists played in carrying them
out. We see how IBM collaborated to digitize and streamline the
Holocaust, contributing to the deaths of six million Jews and millions
of others. We recall the internment of Japanese Americans during the
Second World War. We recognize that mass deportations precipitated the
very atrocity the word genocide was created to describe: the murder of
1.5 million Armenians in Turkey. We acknowledge that genocides are not
merely a relic of the distant past—among others, Tutsi Rwandans and
Bosnian Muslims have been victims in our lifetimes.

Today we stand together to say: not on our watch, and never again.

And they didn't stop there. They've done what engineers do best,
they built something: a platform to allow other people to add their
names. The list currently stands at 1239 people with new pull
requests (the process by which people request to be added) happening so
fast that they literally have had problems keeping up.

What's more, the whole thing is being developed in the open and you can
watch the process unfold. Just yesterday afternoon I was following
this ticket
where they were debating how to solve the onslaught of applicants and
introduce some uniformity for scale. The software chosen was Free,
Open, and conforming to a universal standard that's easy to follow.

What's exciting about this, for me at least, is that this could very
well be the beginning of a Code of Ethics for software engineering:
developed in the open by ourselves, in an effort to operate as a
community for the greater good.

Ethics in engineering is nothing new of course. The
Ritual of the Calling of an Engineer
has been a thing in Canada since 1922 in an effort to make sure that, in
the interest of the Public Good, engineers who built bridges would
adhere to ethics rather than employer directives.

Historically however, software engineering has been a Wild West of
people doing whatever they want, with zero focus on the ethics of what
we're doing. It's my hope that if anything comes out of a Trump
presidency, a sense of responsibility for our actions should be it.

There's a lot of potential here. In an ideal world, I'd like to see
companies and Free software projects adopting a policy of only
collaborating with engineers who have signed the pledge: a simple
declaration that we are thinking people with moral compasses who are
responsible for our actions. In much the same way that companies,
conferences, and projects have codes of conduct, I think it's time that
we acknowledge that ethics should be an integral part of what we do.

This is just one project though, and a rather US-centric one at that, so
I'm not sure it has the legs required to get us to where I think we need
to be, but it's a start, and I'm absolutely thrilled that we're
finally having this conversation.

The Burn It All Down Candidate

The outcome of the US presidential election was easily predicted. I say this
because I was sure Trump would win as far back as May of this year. There
seems to be a great many people still entirely surprised by the outcome though, so I
thought I'd write down my reasoning.

There were two primary factors in Trump's taking of the White House. I'll
deal with the minor reason first.

Sexism

It's still easier for most people to vote for a black man than it is for a
white woman. The reasons for this are long and horrible, but this reality
hasn't changed much in the last few thousand years. A woman may have a better
chance now of winning an election, but sexism continues to stack the deck
against every female candidate in most of the world.

It's a real problem, but I don't think it's nearly enough to explain why
Clinton lost in an election that was the Left's for the taking.

The Establishment

Trump didn't win this election, Clinton lost it because she couldn't convince
people to show up. Take a look at this chart
showing the voter turnout over the last few elections (credit to dinoignacio via reddit).
Trump rode the coattails of the Republican base who vote red regardless of the
candidate, but the Left barely represented, begging the question: why not?

To answer this you need only look at the Bernie Sanders campaign. Here was a
candidate who called for an end to money in politics, real socialised
medicine, a focus on the environment, and on returning industry to the country.
His focus on the future appealed to young people, his record on social policy
encouraged the base, and his rhetoric on taking care of people hit hardest
resonated with everyone who has been hurting over the last few years.

That last group is what matters because that last group is HUGE.

The truth is that in most Western nations, the US included, austerity and the
right-wing have been at war with the working class for decades. In countries
without an adequate social safety net (like the US), it's entirely common to
have a household with two working parents and two children, and still not
have enough to make ends meet. For those families just squeaking by, they live
in fear of one of them losing their job, at which point they are literally
homeless.

These people are angry, and they're scared, and the best that the Left
could drum up was a woman whose dynastic name practically begs the spectre of
corruption and hereditary rule. Clinton is the embodiment of "politics as
usual" handed to a nation of people desperate for change.

At his core, Obama wasn't much different from Clinton in the ideals that
capitalism somehow equals freedom, but importantly on the surface Obama was
inspirational: the first black president, a Democrat who talks like a Kennedy.
That man could have read the phone book to the public and the world would still
have fallen in love with him.

But after eight years with him at the helm, the people are still scared and
angry. They've barely survived a banking crisis that crippled the planet and
saw not one rich white banker convicted. There are riots on the streets
spurred by cops with military hardware murdering cvilians. They're still
living paycheque to paycheque and the only explanation they're getting is from
the Orange Beast on the right who is insisting that the immigrants are to blame
for everything.

This was a hard sell, but the American people were ripe for a real shift in
policy. So what did the Democratic party do? They sabotaged the Sanders campaign
and assumed that fear of the damage Trump could do would be enough to get
the plebs to vote blue.

The people are tired, angry, and scared. They want an end to corruption, to
a government that doesn't understand or even hear their plight. They were told
repeatedly by that establishment that a vote for Trump would be insane, that it
would burn everything down. The
trouble is, that's what the people want. The people want a revolution. They
want to see muderous cops and "banksters" behind bars, they want an end to
foreign wars, and they want coprorations out of politics.

This is a failure of the Left to give the people what they want. The left
could have run an inspirational candidate, one who wanted the same thing 70% of
Americans wanted. This was the opportunity to capture the White House, the
Senate, Congress, and the Supreme Court for that revolutionary vision of the
future.