For testing purpose I installed the clock demo which is part of the
Embedded Artists repository.
Of course I wanted to do more than displaying the time on this little box.
I also wanted to take advantage of the button I had integrated.

One idea was to create a small web server so that I could receive and display
messages. The application would basically:

display the time (every minute)

when receiving a message, stop the clock and display the message

when the button is pressed, start the clock again

I don't know about you, but this really makes me think event loop!
I learnt asynchronous programming with Dave Peticolas Twisted Introduction
a few years ago. If you are not familiar with asynchronous programming, I really recommend
it. I wrote a few applications using Twisted but I haven't had
the opportunity to use asyncio yet. Here is a very good occasion!

asyncio

REST API using aiohttp

There are already several asyncio web frameworks to build an HTTP server.
I decided to go with aiohttp
which is kind of the default one.

Using this tutorial I
wrote a simple REST API using aiohttp. It uses JSON Web Tokens which is
something else I have been wanted to try.

Raspberry Pi GPIO and asyncio

An alternative is the pigpio library
which provides a daemon to access the Raspberry Pi GPIO via a pipe or socket interface.
And someone (Pierre Rust) already created an aysncio based Python client
for the pigpio daemon: apigpio.

Exactly what I needed!
It's basically a (incomplete) port of the original Python client provided
with pigpio, but far sufficient for my need. I just want to get a
notification when pressing the button on top of the screen.

Stop the clock and display a message

When receiving a message, I first cancel the clock background task and
send the messages to the e-paper display using ensure_future so that
I can return a json response without having to wait for the message to be
displayed as it takes about 5 seconds:

Running on the Pi

You might have noticed that I used some syntax that is Python 3.6 only.
I don't really see myself using something else when starting a new project
today :-)
There are so many new things (like f-strings) that make your programs look
cleaner.

On raspbian, if you install Python 3, you get 3.4... So how do you get Python 3.6 on
a Raspberry Pi?

On desktop/server I usually use conda. It makes it so easy to install
the Python version you want and many dependencies.
There are no official installer for the armv6 architecture but I
found berryconda which is a
conda based distribution for the Raspberry Pi! Really nice!

I could have gone with berryconda, but there's one thing I wanted as well.
I'll have to open the HTTP server to the outside world meaning I need
HTTPS. As mentionned in another post, traefik makes
that very easy if you use docker. So that's what I chose.

I created 3 containers:

traefik

pigpiod

aiolegomac

traefik

There are no official Traefik docker images for arm yet, but an issue is currently opened.
So it should arrive soon!

pigpiod

For pigpiod, I first created an image based on arm32v6/alpine but I noticed I couldn't send
a SIGTERM to the daemon to stop it properly... I'm not sure why. Alpine being based on musl instead
of glibc might be the problem. Here is the Dockerfile I tried:

What about the EPD driver?
As it uses libfuse to represent the e-paper display as a virtual directory of files,
the easiest was to install it on the host and to mount it as a volume inside the docker
container.

This will install docker and the EPD driver, download the aiolegomac repository, build the 3 docker images
and start everything.

Building the main application docker image on a Raspberry Pi Zero takes quite some time.
So be patient :-) Just go and do something else.

When the full playbook is complete (it took about 55 minutes for me),
you'll have a server with HTTPS support (thanks to Let's Encrypt) running on the Pi. It's displaying
the clock every minute and you can send messages to it!

Client

HTTPie

To test the server you can of course use curl
but I really like HTTPie. It's much more user
friendly.

requests

So let's write a small script to send messages to our server.
We'll store the server url and username to use in a small yaml configuration file.
If we don't have a token yet or if the saved one is no longer valid,
the script will retrieve one after prompting us for a password.
The token is saved in the configuration file for later use.

The following script could be improved with some nicer error messages
by catching exceptions. But it does the job:

Why did I only write a command line script to send messages and no web interface?
Don't worry, that's planned! I could have used Jinja2. But I'd like to try a javascript framework.
So that will be the subject of another post.

As the title suggests, it describes different ways to run a flask application over HTTPS.
I have been using flask for quite some time, but I didn't even know about
the ssl_context argument. You should definitively check his article!

Using nginx as a reverse proxy with a self-signed certificate or Let’s
Encrypt are two options I have been using in the
past.

If your app is available on the internet, you should definitively use
Let's Encrypt. But if your app is only supposed to be used internally
on a private network, a self-signed certificate is an option.

I then discovered traefik: "a modern HTTP reverse proxy
and load balancer made to deploy microservices with ease". And that's
really the case! I've used it to deploy several applications and I
was impressed. It's written in go, so single binary. There is also a tiny docker
image that makes it easy to deploy. It includes Let's Encrypt support (with automatic renewal),
websocket support (no specific setup required)... And many other features.

A simple example

I created a dummy example just to show how to run a flask application over
HTTPS with traefik and Let's Encrypt.
Note that traefik is made to dynamically discover backends. So you usually
don't run it with your app in the same docker-compose.yml file. It usually
runs separately. But to make it easier, I put both in the same file:

Traefik requires access to the docker socket to listen for changes in the
backends. It can thus automatically discover when you start and stop
containers. You can ovverride default behaviour by using labels in your
container.

Supposing you own the myhost.example.com domain and have access to ports 80 and 443
(you can setup port forwarding if you run that on your machine behind a
router at home), you can run:

I've been playing with some Raspberry Pis before but only with
software.
I have been willing to fiddle with hardware for some time.
This was the perfect opportunity!

LEGO Digital Designer

I decided to try to make my own LEGO Macintosh based on Jannis work.
His blog post is quite detailed with even a list of links with all the
required components.

But I quickly realized there were no LEGO building instructions...
I thus created my own using LEGO Digital Designer, which was fun.
Looking at the pictures on Jannis flickr album
helped a lot. But having an exact idea of the screen size wasn't easy on
the computer. So I also built a small prototype of the front part to get a
better idea. For that I had to wait for my e-paper display.

One modification I wanted to do was to use 1U width lego on the side of
the display to require less drilling.
I also wanted to check if it was possible to use the button located on top
of the display.

E-paper display

I e-mailed them and I was granted to pick up my order at their office!
A big thanks to them!

Raspbery Pi Zero W

The Raspberry Pi Zero W comes with Wifi which is really nice.
It does not come with the soldered GPIO header. I was starting to look at
existing soldering iron when I discovered this GPIO Hammer Header:

No soldering required!
I used the installation jig and it was really easy to install.
There is a nice video that explains how
to proceed:

Connecting the display to the Pi

Based on Jannis article I initially thought it wasn't possible to use a ribbon
cable (due to space), so I ordered some Jumper Wires.
I connected the display to the Pi using the serial expansion
connector as described in his blog post.
It worked. With the demo from embeddedartists, I managed to display a nice cat picture :-)

I then realized that the serial expansion connector didn't give access to
the button on top of the display. That button could allow some
interactions, like changing mode, which would be nice.
According to my prototype with 1U width lego on the side, using a ribbon cable shouldn't actually be
an issue. So I ordered a Downgrade GPIO Ribbon Cable for Raspberry Pi.

It required a little drilling on the right side for the cable to fit. But
not that much. More is needed on the left side to center the screen.
Carried away by my enthusiasm, I actually cut a bit too much on the left side
(using the dremel was fun :-).

Everything fitted nicely in the lego case:

Button on top

With the ribbon cable, the button on top of the display is connected to pin
15 on the Raspberry Pi (BCM GPIO22).
The ImageDemoButton.py part of the demo
shows an example how to use the button to change the image displayed.

Using my small prototype, I planned a small hole on top of the case. I thought I'd have to fill
the brick with something hard to press the button. The 1x1 brick ended fitting perfectly.
As shown on the picture below, the side is exactly on top of the button.
I added a little piece of foam inside the brick to keep it straight.

Pi configuration

Jannis article made me discover resin.io,
which is a really interesting project. I did a few tests on a Raspberry Pi 3 and it was a nice experience.
But when I received my Pi Zero W, it wasn't supported by resinOS yet... This isn't the case anymore!
Version 2.0.3 added support for the wifi chip.

Anyway, as Jannis already wrote about resinOS, I'll describe my tests with Raspbian.
To flash the SD card, I recommend Etcher which is an open source project by the same resin.io.
I'm more a command line guy and I have used dd many times. But I was pleasantly surprised.
It's easy to install and use.

One more thing

There isn't much Python in this article but the Pi is running some
Python code.
I couldn't resist putting a Talk Python To Me
sticker on the back :-)
It's really a great podcast and you should definitevely give it a try if
you haven't yet.
Thanks again to @mkennedy for the stickers!

Next

I didn't build this LEGO Macintosh to use it as a simple clock :-)
I have a few ideas. I'll start with a small web server so that I can receive and display messages.
That will be the subject of another blog post!

I've been using Docker for some time now.
There is already a lot of documentation available online but I recently
saw the same "anti-patterns" several times, so I thought it was worth writing a post about
it.

Avoid invalidating the cache

It's actually an example I have seen several times online.
This looks fine, right?

The problem is that the COPY . /app command will invalidate the cache as
soon as any file in the current directory is updated.
Let's say you just change the README file and run docker build again.
Docker will have to re-install all the requirements because the
RUN pip command is run after the COPY that invalidated the cache.

The requirements should only be re-installed if the requirements.txt
file changes:

Install Homebridge and dependencies still following this page.
Note that I had a strange problem here. The npm command didn't produce
any output. I found the same issue on stackoverflow
and even an issue on github.
The workaround is just to open a new terminal...

Adding Homebridge to iOS

Homebridge and the Home Assistant plugin are now running.
Using the Home app on your iOS device, you should be able to add the accessory "Homebridge".
See Homebridge README for
more information. You will need to enter the PIN code defined in your
config.json file.

You should then see the Homebridge bridge on your device:

And it will automatically add all the accessories defined in Home Assistant!

You can now even use Siri to control your devices, like turning ON or OFF the TV VPN.

Note that I renamed the original switch to make it easier to pronounce.
As described in the README,
avoid names usually used by Siri like "Radio" or "Sonos".

That's it! Homebridge is really a nice addition to Home Assistant if you have some iOS devices at home.

I don't run source activate myapp but just use ENV to update the PATH
variable. There is only one environment in the docker image. No need for the extra
checks done by the activate script.

With this Dockerfile, any command will be run in the myapp
environment.

Just a few additional notes:

Be sure to only copy the file environment.yml before to copy the full
current directory. Otherwise any change in the directory would
invalidate the docker cache.
We only want to re-create the conda environment if environment.yml
changes.

Both ways are not super easy and fast to access.
A while ago, I wrote a small Flask web
application to change some settings in my router. The application just
allowed to click on a button to run a script via ssh on the router.

So I could write a small webapp to do just that.
But I recently read about Home Assistant. It's
an open-source home automation platform to track and control your devices
at home. There are many components available, including Command Line
Switch which
looks exactly like what I need.

The Raspberry Pi is a popular device to install Home Assistant.
But my Turris Omnia is quite powerful for a router with
1 GB of RAM and 8 GB of flash. It's time to use some of that power.

From what I read,
there is an openWrt package of Home Assistant.
I couldn't find it in the Turris Omnia available packages.
Anyway, there is another feature I wanted to try: LXC Containers. Home Assistant is a Python
application, so it's easy to install in a linux container and would allow
to easily keep the version up-to-date.

So let's start!

Create a LXC container

As described here, you can
create a LXC container via the LuCI web interface or via the command
line:

Install Home Assistant

Next, we just have to follow the Home Assistant installation instructions.
They are well detailed. I'll just quickly repeat them here to make it
easier to follow but you should refer to the official page for any update:

The LXC container is just like another computer (a virtual one) on the local network.
To access the router, we have to ssh to it. For this to work without
requesting a password, we have to generate a ssh key and add the public
key to the authorized_keys file on the router:

homeassistant@homeassistant:~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/homeassistant/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/homeassistant/.ssh/id_rsa.
Your public key has been saved in /home/homeassistant/.ssh/id_rsa.pub.

Copy the content of /home/homeassistant/.ssh/id_rsa.pub to
/root/.ssh/authorized_keys (on the router not inside the container).

With this configuration, the switch will always be off when you restart
Home Assistant. It won't know either if you change the state using the
command line or LuCI web interface.
This can be solved by adding the optional command_state line. The
command shall return a result code 0 if the switch is on.
The openvpn init script on the Turris Omnia doesn't take "status" as
argument. An easy way to check if openvpn is running is to use pgrep.
Our new configuration becomes:

Make it easy to access Home Assistant

There is one more thing we want to do: assign a fixed IP to the container.
This can be done like for any machines on the LAN via the DHCP and DNS settings in LuCI interface.
In Static Leases, assign a fixed IP to the container MAC address.

Now that the container has a fixed IP, go to http://<container IP>:8123 and
create a bookmark or add an icon to your phone and tablet home screen.
This makes it easy for anyone at home to turn the VPN on and off!

I already spoke about installing OpenVPN on a Raspberry Pi in another blog
post.

I only connect to this VPN server to access content that requires a french IP address.
I use OpenVPN Connect App on my iPad and Tunnelblick
on my mac.
It works nicely but how to use this VPN on my Apple TV 4?
There is no VPN client available...

End of last year I finally received my Turris Omnia that I supported on Indiegogo.
It's a nice router running a free operating system based on
OpenWrt with automatic updates.
If you haven't heard about it, you should check it out.

Configuring OpenVPN client on OpenWrt

Installing an OpenVPN client on OpenWrt is not very difficult.
Here is a quick summary.

Install openvpn-openssl package (via the
webinterface or the command line)

I already have a custom client config that I generated with Ansible in
this post.
To use this config, create the file /etc/config/openvpn:

If you run /etc/init.d/openvpn start with this config, you should connect successfully!
All the traffic will go via the VPN. That's nice but it's not what I want.
I only want my Apple TV traffic to go via the VPN. How to achieve that?

Source based routing

I quickly found this wiki page to implement source
based routing. Exactly what I want. What took me some time to realize is
that before to do that I had to ignore the routes pushed by the server.

With my configuration, when the client connects, the server pushes some
routes among which a default route that makes all the traffic go via the
VPN:

Ignoring the routes pushed by the server can be done with the --route-noexec option.
I tried to add option route_noexec 1 to my /etc/config/openvpn file
but it had no effect. It looks like that when using a custom config, you
can't add other options there. You have to set everything in the custom
config. I added route-noexec to my /etc/openvpn/myclientconfig.ovpn file and it worked!
No more route added. No traffic sent via the VPN.

We now have to add those scripts to the client config.
Here is everything I added to my /etc/openvpn/myclientconfig.ovpn file:

# Don't add or remove routes automatically
# Source based routing for specific client added in up script
route-noexec
# script-security 2 needed to run up and down scripts
script-security 2
# Script to run after successful TUN/TAP device open
up /etc/openvpn/upvpn
# Call down script before to close TUN to properly remove the routing
down-pre
down /etc/openvpn/downvpn

Notice that the machine IP address that we want to route via the VPN is
hard-coded in the the upvpn and downvpn scripts.
This IP shall be fixed. You can easily do that by associating it to
the required MAC address in the DHCP settings.

The tunnel remote IP is automatically passed in parameter to the up and
down scripts by openvpn.

If we run /etc/init.d/openvpn start with this config, only the traffic
from the 192.168.75.20 IP address will go via the VPN!

Run /etc/init.d/openvpn stop to close the tunnel.

Conclusion

This is a nice way to route traffic through a VPN based on the source IP
address.

You can of course use the router webinterface to stop and start openvpn.
In another post,
I'll talk about an even more user friendly way to control it.

I have a Doxie Go scanner and I scan all the documents I receive in paper.
That's nice, but it creates another problem. All the resulting PDF files have to be named, organized and stored...
Doing that manually is boring and time consuming. Of course that's something I want to automate!

I even bought Hazel a while ago.
It's a nice software that monitors files in a folder and performs specific instructions based on the rules you defined.
It works well but I felt a bit limited and I thought I could probably write something more tailored to my use case.
And that would be more fun :-)

A quick solution I found was to run pdftotext using subprocess.
I looked at PDFMiner, a pure Python PDF parser but I found pdftotext
output to be more accurate.
On MacOS, you can install it using Homebrew:

"Adobe Acrobat PDF Files\nAdobe® Portable Document Format (PDF) is a universal file format that preserves all of the fonts, formatting, colours and graphics of any source document, regardless of the application and platform used to create it. Adobe PDF is an ideal format for electronic document distribution as it overcomes the problems commonly encountered with electronic file sharing. • Anyone, anywhere can open a PDF file. All you need is the free Adobe Acrobat Reader. Recipients of other file formats sometimes can't open files because they don't have the applications used to create the documents. PDF files always print correctly on any printing device. PDF files always display exactly as created, regardless of fonts, software, and operating systems. Fonts, and graphics are not lost due to platform, software, and version incompatibilities. The free Acrobat Reader is easy to download and can be freely distributed by anyone. Compact PDF files are smaller than their source files and download a page at a time for fast display on the Web.\n\n• •\n\n• •\n\n\x0c"

This works quite well. The layout is not respected but it's the text that matters. It would be easy to define some regex to define rules based on the PDF content.

This could be the first step in naming and organizing the scanned documents.
But it would be nice to have an interface to easily search in all the files.
I've already used MongoDB full text search in a webapp I wrote and it worked well for my use case.
But I read about Elasticsearch and I always wanted to give it a try.

{'_id': 'AVhvJMC6IvjFWZACJU_u',
'_index': 'my_index',
'_source': {'attachment': {'author': 'cdaily',
'content': "Adobe Acrobat PDF Files\n\nAdobe® Portable Document Format (PDF) is a universal file format that preserves all\nof the fonts, formatting, colours and graphics of any source document, regardless of\nthe application and platform used to create it.\n\nAdobe PDF is an ideal format for electronic document distribution as it overcomes the\nproblems commonly encountered with electronic file sharing.\n\n• Anyone, anywhere can open a PDF file. All you need is the free Adobe Acrobat\nReader. Recipients of other file formats sometimes can't open files because they\ndon't have the applications used to create the documents.\n\n• PDF files always print correctly on any printing device.\n\n• PDF files always display exactly as created, regardless of fonts, software, and\noperating systems. Fonts, and graphics are not lost due to platform, software, and\nversion incompatibilities.\n\n• The free Acrobat Reader is easy to download and can be freely distributed by\nanyone.\n\n• Compact PDF files are smaller than their source files and download a\npage at a time for fast display on the Web.",
'content_length': 1073,
'content_type': 'application/pdf',
'date': '2000-06-28T23:21:08Z',
'language': 'en',
'title': 'This is a test PDF file'}},
'_type': 'my_type',
'_version': 1,
'found': True}

Of course Elasticsearch allows much more complex queries. But that's something for another time.

One interesting thing is that by printing the content, we can see that even the layout is quite acurate! Much better than the pdftotext output:

In [16]:

print(doc['_source']['attachment']['content'])

Adobe Acrobat PDF Files
Adobe® Portable Document Format (PDF) is a universal file format that preserves all
of the fonts, formatting, colours and graphics of any source document, regardless of
the application and platform used to create it.
Adobe PDF is an ideal format for electronic document distribution as it overcomes the
problems commonly encountered with electronic file sharing.
• Anyone, anywhere can open a PDF file. All you need is the free Adobe Acrobat
Reader. Recipients of other file formats sometimes can't open files because they
don't have the applications used to create the documents.
• PDF files always print correctly on any printing device.
• PDF files always display exactly as created, regardless of fonts, software, and
operating systems. Fonts, and graphics are not lost due to platform, software, and
version incompatibilities.
• The free Acrobat Reader is easy to download and can be freely distributed by
anyone.
• Compact PDF files are smaller than their source files and download a
page at a time for fast display on the Web.

Sending the file directly to Elasticsearch is nice, but in my use case, I'd like to process the file (change its title, move it to a specific location...) based on its content. I could of course update the document in ES after processing it.

It might be better in some case to decorelate the parsing and processing from the indexing.
So let's check how to use Tika from Python.

Tika-Python makes Apache Tika available as a Python library.
It can even starts a Tika REST server in the background, but this requires Java 7+ to be installed.
I prefer to run the server myself using the prebuilt docker image: docker-tikaserver.
Like that I have control of what is running.

This is a test PDF file
Adobe Acrobat PDF Files
Adobe® Portable Document Format (PDF) is a universal file format that preserves all
of the fonts, formatting, colours and graphics of any source document, regardless of
the application and platform used to create it.
Adobe PDF is an ideal format for electronic document distribution as it overcomes the
problems commonly encountered with electronic file sharing.
• Anyone, anywhere can open a PDF file. All you need is the free Adobe Acrobat
Reader. Recipients of other file formats sometimes can't open files because they
don't have the applications used to create the documents.
• PDF files always print correctly on any printing device.
• PDF files always display exactly as created, regardless of fonts, software, and
operating systems. Fonts, and graphics are not lost due to platform, software, and
version incompatibilities.
• The free Acrobat Reader is easy to download and can be freely distributed by
anyone.
• Compact PDF files are smaller than their source files and download a
page at a time for fast display on the Web.

Not sure why we get the title of the PDF inside the content.
Anyway the text is extracted properly and we even get a lot of metadata:

GitLab on Synology

I installed GitLab CE on a Synology RackStation RS815+ at work.
It has an Intel Atom C2538 that allows to run Docker on the NAS.

Official GitLab Community Edition docker images are available on Docker Hub.
The documentation to use the image is quite clear and can be found here.

The ports 80 and 443 are already used by nginx that comes with DSM.
I wanted to access GitLab using HTTPS, so I disabled port 443 in nginx
configuration. To do that I had to modify the template
/usr/syno/share/nginx/WWWService.mustache and reboot the NAS:

The port 22 is also already used by the ssh daemon so I decided to use
the port 2222. I created the directory /volume1/docker/gitlab to store
all GitLab data. Here are the required variables in the
/volume1/docker/gitlab/config/gitlab.rb config file:

Hmm... Not super useful error...
I thought about publishing port 4567 in docker, so what is happening?
After looking through the logs, I found /volume1/docker/gitlab/logs/nginx/gitlab_registry_access.logi. It's empty...
Let's try curl:

$ curl https://mygitlab.example.com:4567/v1/users/
curl: (60) Peer certificate cannot be authenticated with known CA certificates
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.

Same error!
Why is that? I can ping mygitlab.example.com and even access nginx on port 4567 (using curl)
inside the docker container...
My machine is on the same network. It can't be a proxy problem. Wait. Proxy?

That's when I remembered I had configured my docker daemon to use a proxy to access the internet!
I created the file /etc/systemd/system/docker.service.d/http-proxy.conf with:

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:8080/"

Reading the docker documentation, it's very clear:
If you have internal Docker registries that you need to contact without proxying you can specify them via the NO_PROXY environment variable

Conclusion

Setting GitLab Container Registry should have been easy but my proxy
settings made me lost quite some time... The proxy environment variables (HTTP_PROXY, NO_PROXY...)
are not taken into account by the docker commands. The docker daemon has to be configured
specifically. Something to remember!

Note that this was with docker 1.11.2. When trying the same command on my Mac with docker 1.12.1, I got a nicer error message: