July 27, 2018

Javascript can now record, manipulate and play sound thanks to a whole range of audio-related API’s. One of the simpler and more useful parts of that is MediaRecorder which can record sound, typically from the users’ microphone.

If myFunc could block (such as on network access), we don’t want to block Javascript’s single thread, so we asked to be notified when the operation is complete.

Javascript added syntactic sugar to make this pattern more pleasant, in two steps.

The first step is adding a Promise object. myFunc would now returns a Promise, and the caller looks like this:

myFunc(value).then(successCallback, errorCallback)

Very similar. An improvement is you can now return this Promise up the call chain, instead of passing down your callbacks, and you can chain then. I sometimes think of Promise as ‘CallbackManager’. It wraps them and gives them a nicer interface.

March 29, 2018

Gardening is my preferred metaphor for software. When I grow a new piece of software, there is a very predictable path I go through as I attempt to manage it’s complexity. It is, in fast-forward, the history of programming language design. That process is short enough to capture in about 400 characters, as two tweets.

Growing software, in two tweets:1. Write `main`. Put your program in there.2. `main` too big? Extract functions. Use global vars.3. Too many functions? Group functions and their data (replacing global vars) into objects.

4. Too many objects? Group objects into packages. Make some methods private.5. Too many packages? Split out a new project, either library or network service.6. Iterate from 2 when other functions grow too big.

May 22, 2017

Learn Better, by Ulrich Bosner is an interesting, valuable book, that is too long. The information would fit comfortably into 50 pages, but you can’t sell a book that short. Amazon is fixing this, but I have a strong preference for paper books over e-books. Anyway, here are my notes.

The core ideas

There are two big ideas in Learn Better:

I. Learning is a skill which you can improve at

Carol Dweck’s Mindset claims that individuals who believe this (versus thinking the ability to learn is fixed and innate) live “a more successful life”. If you picked up a book called “Learn Better”, I’m guessing you already think you can learn better, so hey, you’re half way there already.

In noticed the same viewpoint in a large study of high-school valedictorians: “The top students readily identified themselves as ‘school smart’. Academic talent, to them, meant the ability to excel at academic learning and school tasks such as note taking, memorization, and testing. Many of them clearly attributed their success primarily to effort rather than ability.”

II. Learning is a generative activity

You don’t learn by loading information from somewhere into your brain. You have to create it. To learn, you must do.

Tangent: I found this interesting because of the similarities with memory. We often think of memory as a video camera, yet recalling a memory is a creative activity. I’d highly recommend Elizabeth Loftus’s The myth of repressed memory which explains this very well.

What does “generative” mean here? What kind of activities must you do to learn?

Take notes in your own words as you read (I’m doing it now!).

Summarize.

Tell other people what you learned (even imaginary other people); this is what we do in programming with the rubber duck.

April 9, 2016

To start a server on a port below 1024 (i.e. 80, 443), you need root permissions or capability CAP_NET_BIND_SERVICE, but you also want most of your server to run unprivileged, reducing your attack surface. The traditional way to achieve this was to start as root, bind the socket, then drop privileges. It’s a hassle, and if you get it wrong it’s a big security hole, so often we just run on an unprivileged port and use something like nginx to proxy. But no longer. There’s a much better way: systemd socket activation.

Here is a Go program that will listen on port 8080 if started manually, or port 80 if run via systemd.

We use NoDelay (TCP_NODELAY) and NonBlocking (O_NONBLOCK) because Go sets these by default on sockets it opens, and we want similar behavior.

Reload the config in systemd (sudo systemctl daemon-reload), and you’re done. If you start it manually it will listen on 8080, if you start it via sudo systemctl start hello it will listen on port 80.

To stop the service you also need to stop the socket. If you don’t stop the socket, incoming connections will auto-start the service (inetd style). systemd will warn you about this.

sudo systemctl stop hello.socket hello

How it works

When systemd starts a process that uses socket-based activation it sets the following environment variables. To check whether we are being started via socket activation we just need to check if one of those environment variables is set.

LISTEN_PID: The process id of the process who gets the sockets. This prevents a child forked from your main process from thinking it is being given some sockets.

LISTEN_FDS: The number of file descriptors (sockets) your process is being given. In our case this will be 1.

Every process gets three standard file descriptors: STDIN=0, STDOUT=1, and STDERR=2. Descriptors given to us by systemd hence start at number 3.

January 12, 2016

Three years ago when I wrote The Joy of Upstart, that was the easiest way to turn your scripts into daemons. Today the future belongs firmly to systemd, so let’s revisit the Upstart post, and update it for systemd.

systemd is here, it is the default on most major distributions including Ubuntu from 15.04, Fedora from 15, and Red Hat from v7, so you can probably use it now. The main exception is Ubuntu LTS, where we need to wait a few more months for 16.04.

Take this python script: /home/myuser/ez_daemon.py

import time
while 1:
print("I'm a daemon!")
time.sleep(1)

We’re going to turn it into a daemon in just two lines. Create /etc/systemd/system/ez_daemon.service:

A systemd file (such as /etc/systemd/system/ez_daemon.service) is called a unit. Paths in units must be absolute, hence /usr/bin/python. systemd caches units. When you change one you must systemctl daemon-reload to use the latest version. systemd will remind you.

A major difference with Upstart is that stdout and stderr are now buffered. Upstart went the extra mile by wrapping our script with a pseudo-tty to prevent the buffering. systemd uses a pipe, which means by default there is a 4k buffer on stdout and stderr. You won’t see anything in journalctl until that buffer fills. We pass -u to python to prevent this buffering, or we could have called sys.stdout.flush() after each print. Note also that you can’t have end-of-line comments, a comment must be on a line by itself.

A particularly exciting part of systemd is the security features enabled by cgroups. The example script includes most of them, which you’ll want to remove as necessary. For example PrivateNetwork wouldn’t be applicable for nginx. Here are full details on systemd security. UPDATE 2018: systemd now has the even safer ProtectSystem=strict and dynamic users.

systemd replaces syslog. As a convenience Ubuntu still runs syslog, messages are available in /var/log/syslog. Fedora no longer runs syslog by default, so /var/log/messages is empty. Instead you use journalctl, which has greatly improving filtering capabilities. Examples in Linux Voice.

Another exciting feature of systemd is socket activation, for example systemd can open port 80 and hand it to your process. You no longer need to start as root, open the port, then drop privileges. This USENIX Login has examples, and is probably the best place to continue reading after this post.

December 11, 2015

Over 170 years ago, on Friday 21st July 1843, at 4 o’clock, Ada Lovelace was working on a mathematics problem, possibly on the first known computer program (it was written that summer).

Specifically she was writing extensive notes on her translation of a paper about Charles Babbage’s Analytical Engine, and she was collaborating closely with Babbage to do so. Had it been built the Analytical Engine would have been the first universal computer. Those notes include the first known computer program (and trace of it’s manual execution), which calculates Bernoulli numbers.

But at 4pm on Friday 21st July, she was stuck. It just wasn’t working. We know because she wrote this letter to Babbage:

My dear Babbage. I am in much dismay at having got into so amazing a quagmire and botheration with these numbers, that I cannot possibly get the thing done today. I have no doubts it will all come out clean enough tomorrow; & I shall send you a parcel up, as early in the day as I can. So do no be uneasy. (Tho’ at this moment I am in a charming state of confusion; but it is that sort of confusion which is of a very bubble nature).

I am now going out on horseback. Tant mieux.

Yours puzzle-pate

That ever happened to you? It’s 4pm on a Friday, and it just isn’t working? You know it will be obvious in the morning. So what did Ada do? She went for a horseback ride!

The tagline of this blog is Solvitas Perambulum, Latin for “solve it as you walk”. I find it particularly charming that 172 years ago, the first computer programmer solved it by going for a horseback ride.

Facebook has a software quality problem. I’m going to try to convince you with three examples. This is important because it demonstrates the time-honored principle that quality matters. In demonstrates it, as Facebook engineers like to say, at scale. I don’t work at Facebook or any competitor, I’m just an observer.

Exhibit A: “iOS can’t handle our scale”

About a month ago a Facebook engineer gave this presentation: iOS at Facebook, which was followed by a discussion on reddit.

The Facebook iOS app has over 18,000 Objective-C classes, and in a single week 429 people contributing to it. That’s 429 people working, in some way, on the Facebook iOS app. Rather than take the obvious lesson that there are too many people working on this application, the presentation goes on to blame everything from git to Xcode for those 18,000 classes.

October 26, 2015

I am porting a server from Java to Go, and need to watch the traffic it receives. The clients include OSX and Windows desktop apps, talking to the server over HTTPS. Here’s how I did that with mitmproxy and iptables. We will setup a transparent proxy.

Instructions are for Ubuntu (arguably the most popular server operating system), but mitmproxy will also run on OSX (anecdotaly the most popular developer laptop) using pf instead of iptables.