Continue reading]]>
A friend of mine, Piotr Klin, is the fastest Masters level cyclist in Poland (as proven during last weekend’s National Championships), the current silver medalist of the Masters World Cup, and one of the fastest people on the bike that I know in person. He is taking on the next challenge this Sunday – the ultimate cycling endurance test, known as the hour record attempt.

In simple words, the hour record attempt is about covering as long distance as possible within one hour, on an indoor bicycle track. Riding a track bike, i.e. fixed gear, no breaks, no water, no eating, around a 250m wooden velodrome — you can already imagine how mentally challenging it can be. And when I tell you that the aim is to beat the national Elite record and world Masters record, which would require Piotr to ride over 50km (within one hour), you should get the idea of what kind of a dreaded nightmare it is.

Piotr Klin in full flight.

In order to succeed in such an attempt, not only the extraordinary physical and mental form is required. Aerodynamics of both the rider and the bicycle play a very significant role too. Piotr is a mechanical engineer and he works on aerodynamics in the automotive industry, so he realizes the importance of a streamlined airflow around the body and the bike very well and is able to utilize his knowledge to improve the results. To the extent of 3D-printing some bike components…

This is a handlebar.

Last but not least, aerodynamics depend heavily on the weather conditions. Namely, on the air density (often marked with a Greek letter rho – ρ), which is a factor of air pressure, relative humidity and temperature. The higher the air density, the more difficult is to travel through the air. Even if its impact is unnoticeable in daily life, in competition, especially at relatively high speed over a long time, it makes a significant difference. Let me just point out that for an hour attempt, just a 2-3% change in air density could alter the resulting distance by up to 300-400m, which can mean winning or losing.

Indoor weather station

Even if the date for the attempt was set and the velodrome was booked a couple of months ago, the precise time of day when Piotr would start was yet to be decided. We developed the simple weather station to figure out the air density at the velodrome and the optimal attempt time.

Requirements

The weather station would measure air pressure, relative humidity and temperature and calculate the air density out of these values.

It needs to do measurements frequently enough so that we’ll be able to nail down the optimal one-hour time window.

It should send the data periodically to the internet to allow analysing it on the fly.

It would be installed on a velodrome, without an easy way to connect to grid power, so it should run on the battery.

There is no LoRaWAN coverage in the area, so it has to use WiFi.

The velodrome WiFi is open (!) and tends to break down under heavy load (e.g. during track race days), and the device shouldn’t lose data due to lack of connectivity.

The velodrome is in Pruszków, Poland. It’s over 100km from where I live, and I’m not visiting it often in summer. The device should then be either pretty bulletproof or easy to fix/reset by the non-technical velodrome staff that I could contact over the phone.

Plus, the hour record attempt takes place on the 19th of August, and it was the end of June when Piotr asked me to do it Given that we’d like to gather a couple of weeks worth of data before deciding on a start time, the device should be ready within days.

Hardware choice

Now I need to stress out that I’m not in any way affiliated with or sponsored by Pycom (ok, that’s not counting the WiPy board I got a long time ago from Jose Marcelino ), but simply put, to make a working prototype within a couple of days, it was the easiest for me to leverage my MicroPython experience and loads of existing code I once wrote to do this new project.

The expansion board also provides a battery connector and the microSD card slot. It drains some extra current for battery level measurement, and that could be optimized on a custom circuit board, but it’s a trade-off for having all the goods mentioned before.

Test circuit.

Fitting the BME280 breakout board was not a simple task, because I got the version with an extra CCS811 sensor that used 7 pins in total, but I finally had the excuse to use the weirdly looking elevated goldpins that I once bought for a no good reason. Needless to say, my RTC module also required some extra wizardry to fit on the expansion board, but at the end of the day, it is possible to connect any two interfaces with a finite number of adapters.

Software implementation

Air Density measurement

I was lucky to find a MicroPython BME280 library that I ported to Pycom MicroPython with just a little effort, so the core functionality was a no-brainer. Well, not. The sensor obviously reports only the raw values of temperature, absolute air pressure and relative humidity. The air density still needs to be calculated. Piotr shared a spreadsheet with me:

Here for example, for values in red cells, the air density is 1.131kg/m^3 (dark green cell).

All I needed to do was to write it in Python and verify that it outputs the correct values. Check.

Data storage and upload

In order to achieve the proper measurement resolution, we decided to capture the data every 10 minutes. Now should the device upload data points one by one to the server, the WiFi would drain the battery in a couple of days at most. Since we didn’t need a real-time information on weather conditions, it was okay to upload the data once a day (or more precisely, once every 144 measurements, which is 24 hours times 6 per hour), and otherwise keep the WiFi radio turned off.

With a native Pycom support for the microSD card, it was a piece of cake to store measurements between uploads in a text file on the card. Already uploaded data points are kept on the microSD card for offline access and for backup purposes.

Timestamping

The problem with uploading once per day is that the device doesn’t know the time between measurements. The real-time clock on Espressif ESP-32 chip (which is the heart of Pycom devices) is not powered during deep sleep, so when device wakes up 10 minutes after last data upload, it’s back to 1970. So when it comes to keeping time during sleep, Espressif is a BYOC (bring your own clock) platform.

The time is synced once per day (during data upload), it is kept in the external RTC module (powered with a separate battery), and it’s used for timestamping intermediate measurements.

Server

No fancy stuff here, I went with good old InfluxDB for storing data and Grafana for visualization. It’s good enough, looks nice and clean and can be up and running within minutes.

Deployment

After proving that prototype works stable at my home, I tested it on the velodrome. Thankfully, it happened during Grand Prix Poland, with lots of people on the track, and, as it turned out, a broken WiFi. I realized that the device tried to connect for too long, then it got reset by a watchdog (that was set to 60s) and it repeated the same thing again until it managed to upload the data successfully (which didn’t happen in a couple of hours).

Trying to connect to WiFi repeatedly over a couple of hours is, of course, deadly to the battery, so I needed another trade-off to save the battery and the data, to some extent. I reworked the code to try to upload the data only once, and on failure don’t repeat, but just store the data on the SD card and move on. This way it would skip a day worth of measurements, but would still work and gather the data, possibly the next day the WiFi would be up again, and I could access the missing data offline by copying them from the SD card.

Conclusions

Piotr will go at 18:00. My work is done and it’s on him now.

PS. It’s August 14th and the battery voltage is at around 3.7V after 38 days. Not bad.

Continue reading]]>
I recently got a Waveshare E-Paper Display, in order to learn more about displays – programming and using them with microcontrollers. I chose specifically the e-paper because of its features like very low power consumption (up to 8mA only when refreshing) and preserving the state when unpowered. The latter can be especially handy for battery-powered sensor devices. Think of an environmental sensor that measures data only once in a while and sleeps in between, but with e-paper, it can display e.g. the last measurement all the time without affecting battery life. For starters, I got the pretty small, 1.54″ display, with three available colours, i.e. black, red, and no colour.

The Waveshare E-Paper Display (EPD) communicates via SPI, uses 6 pins for communication (which with power supply makes 8 in total). It comes with libraries for Arduino, Raspberry Pi (in Python and C++) and STM32, so you’re all set on these platforms. For other environments, well, tough luck.

MicroPython implementation

The Python library for Raspberry Pi uses PIL (Python Imaging Library) for displaying text and images. It’s awesome because it greatly simplifies the implementation. PIL is however not available in MicroPython, so I had to resort to drawing text using fonts defined in the code as arrays.

Full font definition is available here. 3×24 = 72 bytes are required for a single character. Now multiply it by the number of printable ASCII characters (95, in the 32-127 range) and you’ll get 6840 items in the list. Even for the small 8-pixel font the list has 760 elements.

Similarly for images, to display an image you’ll need a full-screen bitmap, which in case of my small 200x200px display is a list with 200×200/8 = 5000 bytes.

This brings up performance problems on all possible fronts: loading a font module is terribly slow and it generates huge memory overhead. Have a look:

Now data is a bytes object with contents of font24.data. Furthermore, bytes is an immutable data structure, which is exactly what we need for font files, and it can allow MicroPython interpreter to optimize it better.

The optimized font file looks like this. Well, you won’t easily consult the individual letters anymore, but that’s the price for it being FAST. Here’s how fast:

More than 4 times faster to load, with roughly a third of an original memory footprint.

If this wasn’t enough, we can go further.

Frozen modules

A lengthy Python module, and especially a huge chunk of immutable binary data is a perfect candidate to be included in the firmware. It’s not a trivial task as it includes building a firmware image yourself, but it’s also not overly complicated. If it’s something you haven’t tried yet, I encourage you to give it a spin.

Note on -j9: it is used to speed up building by parallelizing the compilation of independent objects. The number specifies the maximum number of jobs that can be run in parallel. As a rule of thumb, you can safely use -j5 on dual-core machines and -j9 on quad-core machines.

Flash the image according to the README.

Now you don’t need your frozen modules’ files on the flash because they’re included in the firmware. Other than that you work with frozen modules the same way as with regular ones, i.e. you still import them using import statement, etc.

Here, the garbage collector is irrelevant because frozen modules are apparently already loaded into RAM together with the whole firmware, but as you can see, the improvement in import time is huge.

To recap the optimizations, we’ve tried:

list data: 36kB RAM, 1666ms to import

bytes data: 14kB RAM, 377ms to import

frozen bytes data: N/A (loaded with firmware), 16ms to import

Also, with a simple test program that draws a couple of shapes, prints a few strings and displays an image, the total available RAM remaining after the program has finished is around 2.4MB (with 2.55MB available at startup after loading firmware), when before optimizations it was at around 1.8MB.

Library features

In the current state, the library supports only the 1.54 inch two-colour black and red Waveshare display, but adding support for other EPDs should be a simple task. I’ll maybe get some other unit to give it a try.

Apart from printing text, you can also draw lines (between any two points, i.e. not limited to horizontal and vertical) or rectangles and circles (both regular and filled).

#GOINVENT!

Drawing images is available via filling the frame buffer with raw image bytes (which is the fastest possible way, especially if you freeze the image data in firmware). It’s not a very straightforward or flexible method though, as you have to convert your image to bytes beforehand. So I added support for loading a Windows-style 1-color BMP file directly. You can even display the bitmap at given coordinates, if it’s smaller than the screen size – this should be helpful when combining images with text.

The library is available from GitHub, and should you need the C/C++ version for plain ESP32, I also did it and it’s in a separate repository (although without the BMP file support). Let me know if you find any of it helpful and of course, contributions are welcome!

Continue reading]]>
After updating the Air Quality Monitor with LoPy4 board, when I finally got it up and running with Wi-Fi, I moved on to working on LoRa connectivity. I previously had little to none experience with LoRa, so it took me some time, but it was totally worth it.

This is a story of how I got the air quality sensor to transmit its data via LoRa, from the balcony of my flat, over LoRaWAN nano-gateway to TTN router in the Netherlands, then to my relay server elsewhere in Europe, and to the database on the Raspberry Pi in the living room of the same flat. Just because it can. Buckle up!

But what’s the big deal with LoRa?

LoRa, as in “Low-Power Long-Range” is a wireless communication technology of relatively small bandwidth, with an effective range of up to several kilometres, or even more in the open area. It uses an unrestricted frequency band of 868MHz for Europe, 915MHz for Americas, Australia and New Zealand, and 470-510MHz for China.

Then, the LoRaWAN is a type of Low Power Wide Area Network (LPWAN), which uses LoRa connectivity. The LoRa-enabled end-devices (nodes) connect to the network (the internet) via LoRaWAN gateways in a similar fashion to mobile phones connecting to GSM network via GSM base stations. I mean, a node isn’t assigned to a specific gateway (like in the case of WiFi, when you connect to the given router), but rather it can use any gateway that’s in its range.

A LoRaWAN gateway has the internet connection and passes the data packets from the device (uplink) to the LoRaWAN router. The router then forwards them to the associated web service. It can also handle downlink messages, i.e. data packets sent from the web service back to the device, in response to received uplink messages.

The LoRa’s bandwidth of only 0.3-50kbps is not suitable for every use case – you wouldn’t certainly be streaming a video camera footage or even transferring files with it. But it’s a great fit for connected devices such as various types of sensors (environmental, agricultural or industrial just to name a few) to exchange tiny bits of data. And, most importantly, as the name suggests, data transfer using LoRa requires significantly less power than other wireless technologies like Wi-Fi or Bluetooth LE. You can get more info about LoRa and LoRaWAN at the LoRa Alliance website.

While LoRa exists without LoRaWAN, as you can just set up a 1-1 connection between LoRa-enabled devices, I’ll focus on LoRaWAN in this article.

My setup

Obviously, in order to make use of LoRaWAN, your node must be in range of a LoRaWAN gateway. This is not the case for me yet, but I’m working on getting one – BTW, it’s not extremely expensive as you can build a decent GW for under €200 – see this document for a nice how-to. Hope is not lost though, as you can actually turn a LoRa device into a so-called nano-gateway. A nano GW has some significant limitations – e.g. it requires a specific configuration for devices connecting to it. It’s therefore not suitable for public use but works just fine for development, or for personal use.

A LoPy4 serving as a LoRaWAN nano-gateway.

Quite a while ago, in November last year, I preordered two LoPy4 boards to evaluate LoRa connectivity myself. I received them mid-January, and for a first test, I wanted to connect my air quality monitor to LoRaWAN. One of the boards would replace the WiPy in the sensor, and the other one would serve as a nano-gateway.

Setting up a nano-gateway is dead simple – it’s well documented and you can get the most up-to-date sample code from Pycom’s GitHub. Replacing WiPy with LoPy4 in the air quality monitor was a bit trickier and deserved a separate blog post, but here it is:

The LoRa-enabled version of the Air Quality Monitor.

Connecting to LoRaWAN network

Once you get the gateway up and running you need to connect it to the LoRaWAN router. There are a couple of public LoRaWAN network operators and The Things Network is arguably the most popular one.

Setting up the nano-gateway

If you’re within LoRaWAN network range and you don’t need to make your own gateway you can skip this step.

Otherwise, after booting up the nano-gateway you need to register it at the TTN website. This is a simple process, already documented in many places, including Pycom docs, so I’ll skip it here. It’s also pretty straightforward and the only thing to remember when doing it for the nano-gateway is to be sure to check the “I’m using the legacy packet forwarder” checkbox.

Creating a TTN Application

To do something actually useful with your data you’ll need to pass it on from TTN to the service that would handle and process it appropriately. This is done by an application, that you create and configure in TTN console:

Setting up the node

The node needs to be registered in the TTN too, and it’s configured at the application scope:

The important thing that you need to choose when configuring a LoRaWAN node is the authentication mode, i.e. how nodes are authorized with the application. Two modes are available: Over-the-Air Activation (OTAA), and Activation by Personalization (ABP). Although there are significant differences between these two, at the basic “my-first-app” level we don’t need to go that deep into details.

For a proof-of-concept connecting to LoRaWAN, there’s an example code for both OTAA and ABP nodes available from GitHub (otaa_node.py and abp_node.py). In my case, I had some troubles connecting the node using OTAA. It might be me, the faulty firmware at that time, or some other reason, but I saw people on Pycom’s forum complaining that they couldn’t make OTAA work with nano-gateway too. Anyways, I ended up going with ABP. It’s as simple as copying the Device Address, Application Session Key and Network Session Key from device settings to the node script, and boom, it’s online!

Communication over LoRa

Once you connect the node to LoRaWAN, the communication is as simple as creating a socket and writing bytes to it. The following MicroPython code does that for LoPy4:

To simplify serializing temperature, I converted it to Kelvins (but adding e.g. 50 would work fine too) to avoid negative values for sub-zero °C data points. Multiplying by 100 allowed me to store temperature and humidity as integers while keeping the resolution of 2 fraction digits.

Handling data at TTN

TTN application allows you to define JavaScript handlers for decoding the payload, converting its fields and validating them before forwarding to your server. These handlers are not required, but they give you an opportunity to convert the binary payload to a pretty formatted JSON file, e.g. like this:

Consuming data

Having preprocessed the raw payload we can pass it on to the destination. This is where TTN app integrations come into play. If you’re fortunate enough to be using one of the listed services, you can take advantage of its dedicated built-in handler:

In all other cases, there’s a generic HTTP Integration that allows you to forward the (optionally prettified) payload to the server of your choice. That was exactly my case, as I needed the data back at home, in the InfluxDB server running on my Raspberry Pi.

As InfluxDB comes with an HTTP API, I technically could set up the integration so that it points directly to the InfluxDB server (it would, however, require a different payload format than JSON). But for a more generic case, when you want to add another integration in future, it’s better to keep the JSON format and forward it to the relay server that would process the data accordingly for all services you support.

This is all the code required to set up a single-endpoint Python web service with Flask, that would read JSON data from POST payload, convert it to InfluxDB format and post it to the database. Welcome to Python!

Putting it on Heroku is another story, but it took me just a couple of minutes with the documentation found online. And it was my first experience with Heroku (yes, apparently I’ve been living in a cave).

HTTP Integration at TTN

This, again, is a no-brainer. Once the web service is up, all you have to do is pass the right URL to the HTTP Integration:

There are additional fields to fill in on that form, such as header fields, including authorization (highly recommended for a production server), but this is the minimal required configuration.

Watch bytes flowing

The HTTP integration with lora-forwarder was the last step. Now, given that the gateway is online, the node is in range, and TTN, Heroku and my Raspberry Pi are up, we should see the data appearing on a graph. Let’s take a closer look at the single payload, sent at 22:18:39 CET.

Here you can see the node data reaching the gateway:

The gateway recognizes the app and forwards the payload accordingly. The app decodes the payload to JSON:

The JSON data, via HTTP Integration, is passed on to lora-forwarder, which posts it to InfluxDB:

Timestamps are in UTC here.

InfluxDB is read by Grafana that displays the data:

You might say “Cool, but why bother?”. Well, this time there’s far more to it than just “because you can”.

Why bother

Power consumption

The LoPy4 board draws approximately 140mA current with wifi enabled. When LoRa is active, with wifi disabled, the current is not bigger than 38mA. This is a huge saving, and that’s what you could expect having learnt that LoRa is a low power network. But this is just the beginning.

If you think of sending 30 bytes of data via wifi, you’d normally assume that it’s gonna take a couple of seconds. After the device turns on the radio, it scans for the known network, joins it, acquires IP address from DHCP, and then it can transmit the payload. It can take as long as 10 seconds, or even more if the router signal is poor.

Now the killer feature is that the node joins LoRaWAN much faster, plus you can actually store LoRaWAN connection state in the non-volatile memory and retrieve it next time you need to send data. In a long run, the node is able to send data immediately after switching on the radio, and the total time when the LoRa radio is on is measured in milliseconds.

Take the air quality sensor as an example. With wifi connectivity, it sent data in 6-measurement batches (every hour) and it ran for 2 weeks on a 2400mAh battery. With LoRa, I’m sending data one by one (every 10 minutes), and after 2 weeks the battery voltage dropped from 4.15V to 3.77V (still well above the 3.7V limit). And the air quality monitor is not the perfect example because with every measurement the sensor alone draws 200mA for around 10 seconds. If it only measured temperature and humidity, I’d expect it to work for years.

Update 20/02/2018:I disconnected the sensor when the battery voltage dropped to 3.71V after 22 days of uninterrupted uptime, which is over 50% longer than when working on WiFi.

Range

If constrained by wifi connectivity, you can’t get more than a couple dozen metres of range for your device, maybe a couple hundred metres in open terrain with very well-matched high-gain antennas on both ends. This is a piece of cake for LoRa, where a range of several kilometres is nothing unusual.

Also, let me remind you that LoRaWAN infrastructure is not required if you’d like to take advantage of LoRa transmission. It’s possible to set up a two-way 1-1 LoRa communication between two LoRa-enabled devices, where you have full control over the amount and format of data being transferred.

Costs

One would argue that for a long-range communication you could use GSM, especially with its IoT-friendly technologies like NB-IoT and LTE-M. In many situations, this is the only option and it’s fine, but it comes at a cost. Firstly, you need the right hardware, and GSM-enabled devices are usually pretty expensive. And then there comes the service fee you need to pay to your GSM operator.

Now compare it to LoRa-enabled chips, that you can get for as little as €15 with no additional recurring costs. For me, the choice is obvious, as long as I can ensure that the node will remain within the range of the gateway/receiver.

The other aspect is that in some cases by switching to LoRa you can decrease the complexity of your device. In my case for example, when sending data one measurement at a time, I don’t need to track exact timestamps of data points anymore – this means I could get rid of the external RTC module. Not only it reduces the overall cost of the device, but it also simplifies the code (which is always welcome), and even improves battery life a tiny bit.

To sum up, after first trials, I am just amazed by LoRa and LoRaWAN technology. The possibilities are enormous if you consider the really long range, fantastic battery life and low overall deployment cost. As LoRaWAN is pretty much a social network at the moment, everyone can contribute to its expansion by setting up a gateway (the real one though, not the nano GW). I’m definitely getting one in the next couple of weeks and I’ll try to make it useful not only for me but possibly for my neighbourhood, by deploying some devices around the building

Continue reading]]>
It’s been a while since I deployed the air quality monitor on my balcony. If you’re reading this, chances are you have seen the first article where I described the lengthy process of getting it to work with the ESP32-based WiPy board, optimizing battery usage, visualizing the data and putting everything in a box. Over 2 months later I’m after several iterations of improving it, fixing issues and trying out new solutions.

Measurements were taken roughly every 10 minutes (controlled by a not-so-accurate built-in ESP32 real-time clock), they were sent over Wi-Fi to InfluxDB running on RaspberryPi in 6-datapoint batches (i.e. approx. every hour).

Data was visualized in Grafana (also running on the RPi).

Battery wars continued

The 700mAh battery that I tested while publishing the previous article eventually lasted just over 5 days and nights, around 130 hours. I’d call it quite an impressive result, given that for every measurement the particulate matter sensor had to run for ~11 seconds, draining up to 200mA current alone. So the device stopped working because the battery couldn’t supply it with a decent voltage anymore, and unfortunately, I found that the battery was completely dead and showing zero voltage.

PCM my ass

This is unexpected and somewhat disappointing, because the battery had the Protection Circuit Module (PCM) and yes, it was mounted there as I could see it, yet it apparently didn’t do the job of protecting the cell from over-discharging. Also, judging from the battery voltage graph, definitely something went wrong in the last couple of hours, but I wouldn’t say such weird behaviour can occur around 3.7V:

Other graphs attached as a proof that the system worked fine all the time during the heavy battery drain…

The dead battery shows 0V on a voltmeter, and that’s because the PCM is now apparently active and prevents it from powering the load. The voltage measured on the PCM module input showed 2.85V, which is far below a healthy 3.7-3.6V and even below the (deadly for Li-Po cells) 3.0V level.

The Big Boy

Not thinking too much about it I went on and replace the battery, this time adding a huge 2400mAh battery in hope that it would survive around 2 weeks on a single charge. In fact, it worked great for over 2 weeks, when on the day 16, at slightly over 3.8V, the device stopped working:

On the bright side, this means >2 weeks of uninterrupted uptime, yay!

At first, I was like “oh ok, the Raspberry Pi Grafana server went offline”. But no, it didn’t. The battery was dead again. With the same symptoms (0V and <3V at the PCM input). This was enough of a motivation to revisit the battery voltage measurement code.

It turned out that with some of the recent firmware updates that I got in the meantime, the attenuation level that I used for the ADC changed from 3dB to 2.5dB. This basically means that all the measurements were higher than the actual battery voltage, so when the graph said it was at 3.8V, it was in fact somewhere lower.

In previous firmware the 1 used to mean 3dB, and now it became 2.5dB. It wasn’t my code after all, as I copied the code from elsewhere, but still, I didn’t bother to carefully read it through and try to understand what’s going on there and this is purely my fault. This is the great example of how not paying enough attention to your code can sometimes cost you real money.

All things considered, if the 2400mAh battery died after 16 days, given the discharging curve similar to that of 700mAh battery, we can safely assume that it would still be in a good condition after 14 days, which is the number I was hoping to achieve when planning the battery consumption for the system. Recharging the device twice a month is still not optimal, but quite acceptable for a home sensor.

MQTT

Mostly to just learn what it is and how it works, I wanted to test out the MQTT (Message Queuing Telemetry Transport) protocol. I went with ThingSpeak IoT platform by MathWorks. Setting up an MQTT channel is a no-brainer task, as well as sending the actual data when you have a dedicated library for that.

The air quality monitor, however, since it mostly just sleeps and becomes active only for a few seconds from time to time, is not the best application for MQTT. As a publish-subscribe-based protocol, MQTT can set up a connection once and keep it alive as the device transmits or receives data. Conversely, the HTTP sends one request per connection and then requires reconnecting for another chunk of data – this way it can drain the battery more than MQTT.

LoPy4

Back in November 2017, I preordered two LoPy4 boards, with LoRa and Sigfox connectivity (in addition to Wi-Fi and Bluetooth), 4MB RAM and 8MB flash memory . The main purpose was to play with LoRa, and the natural first step – to connect the air quality monitor to LoRaWAN. When I received the boards a couple of weeks ago I started off by replacing the WiPy with LoPy4 in the device. Which turned out tougher than I would expect.

Running short of GPIOs

Both LoPy4 and WiPy (and all the other Pycom development boards) have the same amount of pins, but some of them are reserved, especially on more sophisticated boards. In case of LoPy4 (and the first version of LoPy), there are three pins reserved for communication with the LoRa chip. This way they can’t be used by the developer. Add to that 6 input-only pins, 2 programming pins, a Wi-Fi antenna-switch pin and overall 9 pins taken by my device’s sensors, it’s getting pretty tight. If I were to add another peripheral, I might have a hard time finding a place for it.

Firmware and RTC issues

Like Pycom staff said on their forum, they got the boards a bit earlier than expected so they started shipping them while still ironing out the software. I could precisely feel it while watching my Guru Meditation errors in the console according to other comments on the forum, these errors could have been caused by importing some modules or calling specific functions/classes (e.g. machine.PWM in my case) too early in the main.py script. It was clearly a firmware bug and it seems to have been fixed in 1.14.0.b1 release.

The other thing I noticed in the LoPy4 was that the real-time clock didn’t persist its state in deep sleep while on battery power. It worked perfectly fine (for an ESP32 RTC ;)) when powered via USB. This was something I heavily relied on, as the script’s workflow looked like this:

sync RTC with NTP server on first boot,

take 6 measurements every 10 minutes,

after 1 hour send the data via Wi-Fi and resync the RTC,

repeat from step 2.

In the current situation, the RTC time was gone after first 10-minute deep sleep. A quick workaround was sacrificing the 10-minute granularity and sending only one data point with mean values of the 6 measurements. But as I wanted to keep the 10-minute resolution, I went with an external RTC module.

I found the DS3231 module relatively cheap and somewhat nicely fitting the expansion board. I’d prefer male pins, but eventually, I sorted it out in some way:

“Any two interfaces can be connected using a finite number of adapters”.

My DS3231 module comes with a battery and has 5 pins: two for power, two for I2C connectivity and one is not connected, yet it still occupies the precious place on the expansion board. I found the DS3231 MicroPython library on GitHub and was able to adapt it to Pycom board fairly easily.

RTC lagging

Once I got the external RTC up and running, I noticed that although it remembered the synced time, it stopped for around 8-10 seconds with every board reset. For the time of reset and boot-up the time wouldn’t have progressed, and after that, it ran fine again, but with this several-second offset. After some investigation, I found out that LoPy4 GPIO pins connected to the power pins for DS3231 were pulled down during reset. In this case, the DS3231 activated some kind of power cut-off circuit that also switched off the onboard battery and was effectively stopping the RTC.

I solved the issue by pulling up the Vin pin of the RTC to LoPy’s 3V3 output. It’s not straightforward to do so on the expansion board, but this solution is good enough for a proof of concept:

Surprise resistor!

LoRaWAN

…And then I managed to connect the board via LoRaWAN to The Things Network. It was an interesting experience, took me almost a whole weekend but now it seems to work reliably for a week already. I’ll better write about it in a separate blog post though because otherwise, no-one would ever finish reading this one.

Custom PCB

Getting tight…

So I initially built the whole device around the expansion board and it was fine for a first prototype, with only two peripherals (the air quality sensor and the T/RH sensor). After adding the RTC, especially with an extra resistor, I thought it was too much and decided to prepare a custom circuit for the sensor.

It’s not overly complicated, basically just connecting peripherals to the LoPy + a 5V power supply for the air quality monitor.

The first, home-made version of the PCB was just a proof of concept, mainly to confirm that I can get rid of the expansion board. I also decided to take things to the next level and manufacture the final boards in a PCB factory. But for now, here it is:

Sorry for the ugly silicone sealing!

It works great, with the exception of battery voltage measurement. Yes, I completely forgot about the voltage divider – with this board I’d need to monitor it manually with a voltmeter every couple of days…

Final PCB project

Apart from the voltage measurement circuit, the other important component missing from the board is the battery charger. I mean, for the device to be shippable and easy to use, it should come equipped with a USB port to charge the battery.

Battery charger circuit

The battery charger circuit includes:

the MAX1811 charger module itself,

the micro USB port,

the LED diode that indicates charging,

the circuit that cuts off the battery from the load during charging, and connects the USB voltage instead. It’s very cool – check out how it works over here.

This adds some significant complexity to the device and especially to the PCB layout, but I’m fine with it as I admire working with a PCB designer. As an Integrated Circuits Design graduate, I can recall a little bit the good old times of working long hours with Cadence Virtuoso, which still, compared to KiCad Pcbnew, is like building with Lego Technics vs Duplo.

The above project got sent to the factory, and while writing this, I’m still waiting for the delivery. I’ll keep you posted as the PCBs arrive, hope I’ll manage to solder the micro USB port correctly

Siri support aside, the button instantly caught my attention. It sounded like a simple project with (somewhat) real-life use case. My friend was in need and I could help him out, possibly sparking his and our other colleagues’ interest in learning more about embedded programming. And, after all, you always wanted a big button on your desk for whatever purpose, admit it.

Scope

I defined the following requirements straight off:

The button would connect to the computer and run a predefined command. Fastlane runs in terminal so I assumed it would first open a system shell and then execute the command.

The wired connection is so 2000, the Wi-Fi connection is too much and not really handy ⇒ Bluetooth Low Energy sounds just fine. I know close to nothing about it (barring mobile apps working with BLE beacons), but that’s fine since I’m going to learn it.

I could write a client app on the OS X that would connect to the button, and allow for configuring its action. I’m normally doing roughly one OS X app every 4 years and it was about time.

Hardware

Remember I said I knew nothing about Bluetooth? That’s true to the extent that I wasn’t sure whether I’d achieve what I wanted with Bluetooth Low Energy, or I needed to go with Bluetooth Classic. I also wanted to cut costs a little bit, so I decided to give it a try with ESP-WROOM-32 microcontroller from Espressif Systems. It’s a very powerful dual-core chip with WiFi, BLE, and Bluetooth Classic connectivity, and it measures a mere 18×25.5mm with integrated PCB antenna included.

Cutting costs pretty aggressively actually.

It has a 1.27mm raster which meant some serious SMD effort just to connect wires to it, so I ordered two chips in case I fail, and planned to solder it to the ESP-32-specific 2.54mm raster adapter.

Fun

For the most crucial part of the device, the Adafruit Massive Arcade Button was the obvious ideal candidate. It features a 10cm diameter dome with an LED backlight, and since I saw it for the first time I was looking for a good enough excuse to finally get it.

Ready, set, go!

One month later when the package from Singapore arrived, I was ready to get my hands dirty. It was a very wise decision to order two chips, as it didn’t take long till I failed at putting the first one on the adapter. I didn’t notice the slight misalignment in the left upper area and made several short circuits, and then unsoldering the chip using decent temperature was too much for my skills.

How to: waste your shiny new ESP-WROOM-32 by soldering it misaligned.

If that was not enough of a bad luck, I wasted the other chip in a similar way just a few moments later. (I thought) I soldered it properly this time (without obviously visible misalignments), but still, something went wrong, as I lost the connection to the chip soon after I powered it up.

Step back. Regroup.

Having failed so miserably I resorted to getting a dedicated development board. I thought I’d do it the right way – start off with the dev board, and having figured out the working prototype, I’d replace it with a bare microcontroller and only the required circuitry. Luckily enough I got an ESP-DevKitC just before the Christmas break.

Prototype

The circuit is really simple – here’s what’s needed:

1x ESP-32 dev board,

1x tact switch,

1x LED,

1x resistor (to limit the LED current).

The LED (and the resistor) could be omitted but since it’s already provided with the Adafruit dome button, let’s make some use of it.

Simple as that.

Working principle

I imagined the big picture the following way:

the button device is a BLE accessory, just like a heart rate monitor,

the computer is a client that connects to the button; it can be configured to run a predefined command when triggered,

when the button is pressed, it notifies the client; it also switches on the LED to indicate a busy state,

the client acknowledges the notification and runs a command,

when the command finishes, the client updates the button so that it turns off the LED and waits to be pressed again.

First run

Like I mentioned earlier, I was new to Bluetooth Low Energy, especially server-side, and then I was also new to ESP-32 and esp-idf development environment. Having spent some quality time coding in MicroPython for Pycom boards, I was pretty upset to realize that at the moment ESP-32 MicroPython still lacks some of the chip’s features, including Bluetooth APIs. This meant that I had to resort to plain C programming in FreeRTOS environment.

Fortunately, I was able to base my work on the GATT Server example of the esp-idf, and adjust it to my needs, removing unnecessary code and adding handlers for the button and the LED.

After some trial-and-error fiddling with both the development board and the OS X app I was extremely happy to see them interact with each other:

Implementation

Server (the button)

Going into details, the button implements the GATT Server, and the computer acts as a GATT client. The Server defines two services:

trigger – with the read-only characteristic with notification capability, sent out when pressing the button,

idle – with write-only characteristic, accepting a boolean value (effectively only the true value) to indicate that the task finished.

Client (the computer)

The Client searches for peripherals with mentioned services. Once found and connected, it sets the idle characteristic to true and requests notifications for the trigger characteristic. In the UI it allows for defining a command (or set of commands) to be passed to the Terminal.app for execution.

So much for the UI.

Now for the tricky part: figuring out when the command finished. Since the app passes control to the external application (Terminal.app), to know when Terminal.app is done running a command, it needs to handle some kind of inter-process communication. Which is not a big deal, at least compared to how to tell the other app to notify my app back.

Inter-process communication in MacOS X

Well, in case of Terminal.app, it runs a system shell so we can easily inject some command into the shell to do a callback for the BuildButton.app. As it comes to the IPC itself, there are several solutions for OS X ranging from a shared file on disk to using Mach ports. I ended up somewhere in between by using the oldie but goodie Apple Script. With its funny, naïve syntax and the ability to expose virtually any behaviour to the app’s scripting interface I believed it was the right tool for the job here.

For opening up the Terminal and running a command, I came up with the following helper script:

on runInTerminal(command)
tell application "Terminal"
activate
set newTab to do script(command)
end tell
return newTab
end runInTerminal

And the command itself had to contain the actual user-defined list of commands, plus a callback to our app:

It tells the app to use FinishCommand, which is a subclass of NSScriptCommand and does the actual job of updating the idle characteristic of the button.

The dedicated button circuit

Since the dev board prototype turned out working fine, I could finally order THE BUTTON, and design a PCB for the circuit. I also decided to give it a last shot with soldering the bare ESP-32 chip to the adapter and I finally got it right

The first version of a circuit contains the following components:

3xAAA battery power supply (4.5V),

3.3V step-up/step-down voltage regulator for the ESP-32,

9V step-up voltage regulator for the LED (I was told it’s for 5-12V but I was short of time and I didn’t want to risk it not working or flashing too dim),

an additional circuitry for the chip to work fine: power supply capacitors, and EN and IO0 pin pull-up resistors,

button interface,

LED interface,

programming interface (UART + VCC/GND).

This is the schematic:

I copied over some part of it (like EN/IO0 and power supply capacitors) directly from the schematic of ESP-DevKitC which is available online. If you take a closer look at my schematic, you’ll notice that it’s highly unoptimized, e.g. it includes two DC voltage regulator circuits: a step-up 9V for the LED and a step-up/step-down 3.3V for the chip. Firstly, it’s a pretty expensive solution, and also the 9V regulator drains about 1mA of constant current when idle, with significant impact on battery life.

Power supply issues

I did some initial bare board testing with the circuit similar to this:

And I couldn’t get it to work because the chip was constantly restarting right after booting up. The console output showed the following message:

Brownout detector was triggered

It means that the power supply can’t deliver high enough voltage to drive the microcontroller. In my case, I was powering it from my computer’s USB port via the USB-UART adapter, which should deliver a solid 3.3V for up to at least around 500mA. I also tried the same with a 3.7V Li-ion battery and the standalone regulated DC adapter with no luck. Then I decided to have a look at the program I was running on the ESP.

When I tried it with a simple hello world example, everything worked fine. But every time I initialized Bluetooth, the brownout detector fired and restarted the device. This must have been caused by a high current peak occurring when powering up Bluetooth. And that’s what the bulk capacitor sitting between power supply lines is supposed to handle – cover the current peaks with its charge. In my case, however, it didn’t work with a capacitor placed on the breadboard. When I later dug into the topic I learned that this capacitor:

should be added as close to the chip power input as possible – apparently, a 20cm wire between the breadboard and the microcontroller is not close enough,

should be of high quality, i.e. low impedance – the regular electrolytic capacitor is actually quite the opposite.

Even trying to place the huge (680uF) electrolytic capacitor so close that it directly touched the power input pins didn’t work. I ended up disabling the brownout detector in my chip, which is obviously not recommended*, but it didn’t cause any unexpected crashes in my case. I still ordered a bunch of low ESR SMD capacitors and hoped it would work on the final PCB.

[*] – you usually risk the undefined behaviour in places where the logic level cannot be determined due to the voltage being too low for a logic one, yet too high for a logic zero

Printed board design

For the PCB, I wanted it to be some kind of a “hat” on top of the ESP adapter board:

Quite a tight fit for a hand-made PCB, if you ask me.

Even though the dome button itself is huge and it needs an even bigger case, I went with a reasonably small form factor, and the final board looked like this:

With help of some velcro and a glue gun I managed to put everything in a box:

And give it a proper test:

Also, with the bulk capacitor soldered just millimeters away from the VCC/GND pins I managed to get rid of the brownout issues!

Recap

I must say that when I thought of doing the button for the first time, I had no idea just how much work it would require. After all, it sounds extremely simple, you press a button and it does one simple thing on the Mac. Obviously the requirement of using plain C language for the ESP32 app had huge impact on my perceived effort of this project. And while the device does its job already and I can showcase it to people and get very positive feedback (again, who doesn’t like big arcade-style push buttons), it’s not yet ready to be left alone in the office for the guys to actually trigger builds with it.

Have a look at the short list of features:

it can connect to the client app and remember it

it, most importantly, can trigger a client app to run a command

it does the pulse/heartbeat LED flashing when the command is run; the heartbeat stops when done, or it can be turned off by pressing the button again while running the command (it effectively resets the button state to idle)

it goes to deep sleep after 20s of inactivity (very short time, to conserve battery); when pressed during deep sleep it starts advertising and if the remembered client connects to it, it sends trigger notification – but this adds a 2-3s delay.

And for bugs, or stuff that’s missing:

The device drains around 1500uA when sleeping, and I bet it’s due to the sophisticated voltage regulators I used for both the chip and the LED circuit. Skipping these regulators should have reduced the deep sleep battery drain, and I’m going to give it a try.

The ESP-32 drains up to 150mA when Bluetooth is active and LED is flashing and it’s a ridiculously high current. I tried optimizing it by e.g. clocking down the CPU, disabling one of the cores, limiting Bluetooth radio power but no luck so far. I don’t use WiFi when the chip is running, and the LED current is less than 10mA. It may be that ESP-32 is simply not the right tool for this job. The Nordic Semiconductor’s nRF51822 would probably work much longer, but would possibly require even more work (…?)

Switching between clients is not sorted out yet – for now it needs the currently paired client app to disconnect (e.g. by quitting the app) in order to connect to the other one. I wanted as minimal UI as possible in the device, so I ended up with just one button. I recently implemented the long press recognizer and I’ll try to add pairing mode there.

And so on and so on… Even if the project is unfinished, it gave me yet another solid lesson of embedded systems. I was able to learn about esp-idf, basics of Bluetooth Low Energy, soldering 1.27mm raster integrated circuits just to name a few. Oh and AppleScript, obviously

Also, you know it was worth the effort, when people want to have such a button right after you showcase it to them. And when you give an awesome talk on improving iOS apps build performance and automating tasks, but the main topic on Q&A is how does the button work?:

I’ll still be tinkering with the button as time permits, trying to make something shippable. The up-to-date code is available on GitHub, for both the button itself and the client app. Stay tuned for more weird stuff!

Why bother?

Base iOS client version 3.5.2 is a build number 2259. Base 2.3 from January 2013 was a build number 270. Roughly 2000 builds in 5 years means roughly 8 builds every week, and if you won’t optimize it to be a streamlined task, it can become a painful experience.

Other than just automating release builds, you might want to perform regular health checks of your project. Continuous integration helps in that by running unit tests and/or automated UI tests suites. But it can be additionally upgraded to collect some interesting data while already making builds.

Common tools

We recently adopted fastlane to automate various tasks around our codebase. I’m not sure if it requires an introduction, but if you were living in a cave just like me, fastlane is a modern, good-looking automation tool, written in Ruby and developed very actively all the time. Not only it contains a great number of built-in actions but it also allows you to define your own actions or plugins that nicely integrate into it.

We run continuous integration using Jenkins, but that’s not sexy at all so I’ll just stop here by saying that we configured Jenkins jobs such way that they simply call fastlane to do all the heavy-lifting.

Uncommon tools

What’s perhaps more interesting, is that while we’re running CI tasks we also gather additional information about the build, like the duration, warning count and code coverage stats. We store that information in InfluxDB – a lightweight time series database with HTTP API. We then visualize the data with Grafana. What is very important – both of these tools are available for free when you self-host them so you can give them a spin any time, and I recommend it. I personally run them at home on my RaspberryPi and use them for some cool stuff.

AppStore release builds

Let’s have a look at the list of AppStore release chores for Base App:

sync submodules

install pods

set desired build version

update build number (check HockeyApp for the latest beta/RC build number and increment it)

go to iTunesConnect; create a new app version

go to Xcode; select AppStore scheme

build and archive

upload to iTunesConnect

locate the Xcode archive

compress debug symbols

upload debug symbols to HockeyApp

compress the whole Xcode archive

upload the archive to Dropbox

It could take around 60-90 minutes to do all these things, possibly a little less if you focus solely on the release job. I mean, while waiting for a release build to compile, and then for various uploads to finish, you oftentimes decide to start a side task in the meantime (like a small bugfix, testing, documentation, facebook, etc.) that you’d still have to interrupt shortly in order to follow up with the release. In case you forget you were releasing an app, the process could extend up to a whole day…

Let’s try to eliminate the human factor. The list of AppStore release chores for Base when using fastlane:

call bundle exec fastlane submit version:X.Y.Z

There’s no point 2.

Oh, and it also fetches the most up-to-date certificates from Apple Developer Portal, I forgot about it because it just automagically happens while I’m moving on to other tasks of the day.

34 minutes? Not bad for a release build and uploading over 300MB of data (ipa file, dSYMs + Xcode archive) on my home internet connection. And all that happened while I’ve already been ~33 minutes into my next tasks*.

Yes, such a release lane (as fastlane calls its user-made scripts) doesn’t come bundled with your copy of fastlane and you’d have to implement it yourself, but most of the time it’s just a matter of using built-in fastlane actions with an occasional Ruby scripting**. Surely a doable thing and definitely worth the effort as it pays off quickly.

For what it’s worth, I published a fastlane Dropbox plugin that you might find useful if Dropbox is part of your release flow (or any other flow you’d want to automate for that matter).

[*] – Pro tip: it’s a good idea to have another copy of your repository for creating AppStore builds (and for other automated tasks in general). This way you’re not blocking your working copy of the repo with fastlane run, so you can freely switch branches, etc.

[**] – If you’re allergic to Ruby, you can also write your Fastfiles in Swift! Read here for more info.

Preparing beta builds

There are significantly fewer steps to complete in order to create a beta build for HockeyApp – and these are a subset of the AppStore lane, so with minimal modifications, we got that case covered too. And even though we used to have a custom-made script for preparing beta builds, being able to replace it with fastlane was very relieving, as it’s one thing less to care about.

Automated unit testing on CI

The elementary task for continuous integration is building the app and running unit tests. With fastlane, it’s as easy as calling its scan action, e.g. like this:

lane :unit_tests do
scan(scheme: "Example")
end

It cleans the project, builds the app for testing and runs tests suite. It’s widely configurable, so if you’re new to fastlane, take a look at the documentation for detailed info on scan. For our case, provided that we prepend it with submodule update and pods install, we’d be good to go, only having to direct Jenkins to fastlane’s output files.

Tracking build duration and warning count

We were setting up fastlane for Jenkins just at the time of our warnings crusade, so we wanted to keep clear visibility into what’s the current warning count and whether pull requests introduce any new warnings. We also gave it a shot with measuring CI build time – although completely unrelated to the local build time (our iOS CI runs on Mac minis that are OH SO SLOW), and prone to significant variations (dependent on the current load of the given Jenkins slave), it could still give some value to us.

Since scan can’t build without running tests, and that’s what we needed for measuring build time, we had to use a lower level xcodebuild action. The flow is as follows:

generate code coverage report using slather, but only on the master branch, because it takes a significant amount of time and we don’t want to do it for every update to every pull request.

Visualizing data

The data stored in InfluxDB is retrieved by Grafana to be visualized as graphs and/or tables:

This must be called a locally consistent data.

As you can see, the build duration may differ wildly on the CI server, even for two builds done next to each other. But one could argue that after neglecting the spikes you can observe some kind of a trend line, and it doesn’t even have to go down to indicate our build time optimization effort, because:

we set up Grafana only close to the end of the build time optimization effort,

we have been adding more code to master branch all the time, so it must have required more time to compile.

On the other hand, the warning count graph does the job perfectly:

You can see the build warnings count on master constantly getting lower, and the rapid slope around 20th of November is the merge of UILocalNotification rewrite – our last huge build health offender.

Also, notice the heart to the left of the graph title. It signifies that the alerting is enabled for this graph. It’s configured so that whenever a data point with warning count greater than the current master occurs, it posts a notification to our HipChat room:

This way the right person could be blamed instantly

Our whole Grafana dashboard is still a work in progress. Although it’s pretty messy without additional filtering and with 2-month time span, you can easily filter by time range, branch, user or version to get the insights you’re interested in:

As a bonus, at the very bottom, there’s a graph of the duration of local Xcode development build done by developers while working (when in the office, because our InfluxDB server is inaccessible from the outside). It uses custom Build Phases of Xcode project to mark the beginning of a build and compute the duration, and then reports the time alongside some metadata to InfluxDB.

The goal here was to get an idea on how long the average build (no matter clean or incremental) takes, and well, the number is close to 30 seconds, but I’m not quite sure what to do with it now. It’s not actionable because the value is acceptable, given the big amount of code. I’d probably try and rework the script to calculate only clean build times.

We also kind of hoped to get the proof that new MacBook Pro compiles significantly faster and so the whole team needs an upgrade, but it doesn’t seem to be the case this time

Recap

Automating common tasks around our codebase was the last remaining task of dealing with increased tech debt. During last two months, we resolved almost 400 warnings, reduced build time by half, started saving some significant amount of work time by offloading common tasks to the appropriate tools and gained much better visibility and control of our codebase health. The work is not by any means finished, but it has actually only started. However, after such a solid initial effort, following up will be much easier.

Thank you for reading through, and I hope it wasn’t a total waste of time!

Continue reading]]>
If I told you Base iOS client takes around 3.5 minutes for a clean build on a decent MacBook Pro (Mid 2015) you would probably say it’s not that bad, considering around 2000 classes and 30+ cocoapods. However, since we haven’t ever worked specifically on build time optimization, we surely could do something to cut it down a bit.

This is the second part of a series of blog posts related to limiting tech debt in Base iOS app. Take a look at the first part if you like.

Measuring build time

To start with, let’s display total build duration in Xcode. Typing the following in terminal:

$ defaults write com.apple.dt.Xcode ShowBuildOperationDuration YES

and restarting Xcode will get you the build time in Xcode’s toolbar after every build. Just like this:

Apparently, it’s an undocumented and unofficial feature that is subject to stop working with any future Xcode release, but so far it’s been serving developers since at least Xcode 5 (according to this discussion on SO). The downside is that duration retrieved this way can’t be logged to a file or exposed in an environment variable, but it’s perfect for controlling the build time as you work on improving it.

Improving build time

Having set the benchmark, you can go through the list of things that can be applied to every project in the pursuit of faster build times.

Look for unused code

As silly as it may seem, your project can contain code that is either no longer used, or obsoleted by new APIs that can do the same but behind the scenes, and also possibly faster. This is true especially the bigger and older your codebase is.

The same holds for external dependencies like cocoapods. Sometimes you stop using a helper library (e.g. UIAlertView+Blocks obsoleted by UIAlertAction) and you forget to remove it from Podfile. It really can happen if the Podfile is long enough

Spot potentially obsolete dependencies

Sometimes you can get rid of a pod because iOS SDK finally implemented the missing functionality (e.g. local notification banners can now be displayed while your app is in foreground, which in most cases looks better than third party toast views).

For the reasons mentioned above, we were able to remove 6 pods out of 37 we’ve originally used – that’s a lot less to compile.

Debug Swift compile time

If you suspect that your Swift code takes too long to compile you can verify it by adding the following Swift compiler frontend flag to Other Swift Flags section in build settings:

-Xfrontend -debug-time-function-bodies

It causes the compiler to output to the build log the time taken to compile each Swift function, just like that:

But you don’t really have to go through the build log to investigate compile times one by one. Here comes the other useful frontend flag:

-Xfrontend -warn-long-function-bodies=150

Setting this flag will generate a build warning for every method that takes more than 150ms to type check. The time limit is obviously just an arbitrary value and you can fine tune it to suit your needs.

Be sure to check it out and read his two blog posts mentioned in the Readme file for more info on compile time problems he was able to debug and fix.

My personal feeling though is that Swift compiler has greatly improved between Swift 2 and Swift 3 and the weirdly long compilation times should be gone now. At least we haven’t really experienced unusual slowness in our Swift 4 codebase.

Experiments with optimization

The whole module optimization provided by Swift compiler is a perfect way to reduce the binary size and speed up execution time for release builds. It can also greatly improve compile time, which is not clearly stated in the WMO blog entry at swift.org.

It is, however, a release build feature and enabling it for debug builds would make debugging virtually impossible. Unless you do it like Cookpad does.

Yusei Nishiyama from Cookpad gave a very inspiring talk at Mobilization 2017 on how they handle development of an iOS app with 100 million monthly active users. He talked about enabling whole module optimization for debug builds to speed up compilation.

First, you need to enable WMO in Swift Compiler’sOptimization Level build settings (which effectively appends -whole-module-optimization flag), and then disable optimizations using -Onone in Other Swift Flags (which doesn’t seem to be affecting the WMO flag). Simple as that.

Here are build times for a clean build of Base app with different optimization levels:

As you can see, enabling WMO makes compilation faster by a factor of 1/3, which is a huge improvement.

Modularizing the code

This is not the simplest task ever, especially if you don’t apply it from the start of the project. Still, it’s a great idea to extract files that:

Creating a new Swift module, step 1.

have no dependencies on Obj-C code,

have no dependencies on other Swift files, or all their dependencies are extracted together with them,

to form separate binaries linked with the main application’s Swift module. This has a couple of advantages:

clearly defined responsibilities – a module, by definition, groups the APIs that have similar purpose; creating a helper module that does X, removes that X from the main module responsibilities list,

better control of the API visibility – you can use open (accessible and subclassable), public (accessible but not subclassable), and internal (not accessible outside a module) access specifiers for all the definitions within a module.

reusability – it’s very simple to copy the whole module between projects and use it elsewhere.

open sourcing – once you’ve prepared a small, self-contained Swift module, you might want to contribute it to the Open Source community,

incremental build time improvement – while the clean build time would remain pretty much unchanged (with a sane number of modules at least), every incremental build won’t touch already compiled modules unless their source has changed. This means fewer files to recompile with every change to the code, and eventually, faster build time.

We created two modules within Base, and I believe the same could be a valid choice for most projects (if you don’t have a better idea):

BaseToolkit – with various helper classes and extensions that are self-contained.

For a proof of concept, modules are 10-20 classes each and around 3500 lines of code in total, but we’ll be adding new code there or migrating old code from the main module as time permits. The whole module optimization can (and should!) obviously be enabled on them like I described above.

Outcome

Going through all the ideas I described and taking appropriate actions, we were able to reduce the clean debug build time by nearly 50% (while adding some new code since October, as I’m writing this in January and testing with Base v3.5.0).

The Jenkins build duration went down from over 12 minutes to below 8 minutes. More insights on that, plus how we’re using fastlane to automate various tasks are coming up in the last part of the series. Subscribe to the blog if you like, and you won’t miss it thanks!

Continue reading]]>
Welcome to a short series of blog posts on how we handled a great deal of ever increasing tech debt at Base CRM iOS app, and used automation to prevent it from growing again. In this introductory part, I’ll give you an overview of our codebase and share a few insights on how we resolved all the compilation warnings we’ve had. Have a read to get a grasp of what can happen when you work hard on new features and limit maintenance work to bug fixes.

We’ve recently released version 3.0 of Base iOS client. A big achievement for us, but let me put it in context so that you can realize it too:

timeline: when I joined Base 5 years ago, we were (already) at version 2.4, and the latest release before 3.0 was 2.57.3. We used to be pretty productive, releasing a new minor version roughly every month, but for over 5 years we’ve been iterating over the same UI and data concept. Now version 3 meant a thorough rework of the core components on both UI and backend side.

workload: the 3.0 release hit the AppStore around 20th Oct 2017, and we started working on it around Christmas 2016. While our iOS team is normally split between several projects running in parallel, this release required the whole team to work together most of the time.

Now that you have the idea how big an effort it was, it makes sense that after the 3.0 release we all got a couple of weeks to recover and do some housekeeping, instead of jumping into new projects right away. This was a perfect time to take a closer look at our codebase, fix old warnings, verify weird project settings, get rid of unneeded code and possibly improve the build system.

The size of the code

For reference, here’s the output of a cloc run on the Base iOS client v3.0.0 tag, excluding pods, submodules and tests code:

Seriously, I’d love to know where’s the D language code

Compilation warnings

Behold the picture, that is worth a thousand words:

No, it doesn’t include pods warnings, just our own code.

Yes, it’s a shame.

It obviously didn’t suddenly grow from 0 to 400-ish overnight. As you know, a number of APIs get deprecated with every major iOS release and it’s not always a straightforward fix. So it gets postponed to after the release, but in case new project kicks in, you end up with no fix at all. Even worse, a couple of times we fixed some of them only to have the pull request wait forever for merging because it touched N+1 places in the app and required a thorough testing, and there was no time to do that because we’re releasing like crazy all the time.

400 warnings not only make Xcode’s toolbar look bad, but it also renders Issue Navigator almost unusable. It’s very hard to scroll through the list of build issues, plus you’re very likely to miss new warnings that you might be just adding. Well, apparently Xcode developers didn’t optimize for that amount of issues, and perhaps this time they’re not to blame (!). I mean, this shouldn’t have happened in the first place.

What’s in that number? The usage of deprecated API, like UIAlertView/UIActionSheet, UISearchDisplayController or UILocalNotification, and legacy frameworks like AddressBook and CoreTelephony accounted for around 70-80% of all the warnings. The rest was other one-off API deprecations, nullability specifiers issues, and minor syntactic inconsistencies. Then the Swift 4 reduced @objc inference, and the trickiest one to fix in a large codebase:

ld: warning: Some object files have incompatible Objective-C category definitions. Some category metadata may be lost. All files containing Objective-C categories should be built using the same compiler.

It boiled down to going through Swift extensions of some Obj-C classes and replacing @objc static var definitions with @objc static func. Some, meaning those included in iOS SDK, or more precisely, those built elsewhere and linked with the project as binaries (i.e. not necessarily built using the same compiler). The tough part was that the offending definitions weren’t pointed out by individual warnings, but they had to be looked up manually.

I don’t want to waste your time writing too much about warnings, especially that most of them should have been fixed 1 or 2 years ago and it’s non-news as of today. The most important point about our warnings crusade though is that after some solid effort we managed to bring the warnings count down to 0! Including the funny last one, that we actually kind of gave up on:

Blast from the past.

The thing is, we don’t even use launch images, as we replaced them with Launch Screen storyboard. We don’t have launch images at all in the project. My bet is that something is wrong with the project file – after all, it dates back to iOS 5… Perhaps we could fix this by creating a new project from scratch and re-adding all the files and settings, but there are no volunteers for that yet, so we ended up adding the placeholder file. It doesn’t even show up on the iPhone 5S, but most importantly Xcode is happy now.

Recap

Fixing all the warnings in the project for sure feels rejuvenating. It improved the quality of the code, but it also made our work easier and more efficient. It’s hard to measure the impact of these changes on the build time because the code changed drastically in some places, but I believe that it has improved in some way (at least for the build from Xcode, because it doesn’t need to display 400 warnings anymore). If it’s not the build time itself, the Xcode UI alone for sure seems faster for the above reason.

Still, build time optimization is the other big topic we tackled once the warnings got resolved, and I’ll write about it in detail in the next post. And back to warnings, we are now able to automatically control build warnings count and ensure we don’t introduce any new warnings to the master branch. I’ll cover it too in the following posts, stay tuned for more!

And this is with an emptymain.pyfile. The printer library itself is quite heavy and once you initialize the printer and set up WiFi connection, you’ll have around 36kB available. This makes printing bitmaps (including QR codes) a bit problematic, especially the way it had been originally implemented in the Adafruit library, i.e. by loading the whole bitmap into memory and sending it to the printer. I skipped this feature for the initial version of MicroPython lib and left it for later.

Now that I have fixed issues around regular text printing, the time has come to do something about bitmaps. I’ll still wait some time for LoPy4 boards, plus it surely is an interesting challenge to try getting it to work on LoPy v1.

The bitmap format

The thermal printer is a rather crude image-printing device when you think of it. In fact, it allows for two colors: black and no color. So, precisely, it’s one color… But because of that, the bitmap can be represented as an array of bits, where 1 means black dot and 0 means blank dot.

The printer’s datasheet defines a couple of different mechanisms for printing bitmaps. The simplest one (that was implemented in the original Python library) can be described as follows:

Send the appropriate command to start printing bitmap

Pass the number of lines (the height of the bitmap)

Pass the number of columns (the width of the bitmap) in bytes (i.e. width in pixels divided by 8)

Send the bitmap data as a sequence of bytes. The printer would add line feeds where appropriate, based on the number of rows.

The example bitmaps for Python library are defined in the code as arrays of integers. A 75x75px Adafruit logo is a 750B array – 75 lines per 10 bytes (required to code 75 dots), while a 135x135px Adafruit QR is over 2kB of raw data. It turns out that loading so many bytes at once into LoPy’s memory is a no-go.

When I dug deeper into the code I realized that my printBitmap() function had been a bit buggy. Once I fixed it I was able to print the smaller bitmap straight away! But it was still failing to allocate memory when trying to load the big QR code bitmap.

Printing from file

I tried streaming a bitmap from a file on disk. I stored the bitmap bytes in a file, and then passed it as an argument to printBitmapFromFile() function like this:

# first two args are width and height in pixels
printer.printBitmapFromFile(135, 135, '/flash/lib/qrcode')

It would open the file, read it in one-line chunks and send the data to printer. This way only a small part of data would be kept in memory at once, effectively allowing for processing the whole bitmap.

In fact, this time I didn’t get a memory allocation problem, but this is how the bitmap was printed:

The QR code as seen by thermal printer – 2017, colorized.

Still suspecting some memory issue, I tried reading only part of the file, limiting the number of lines passed as a second argument to printBitmapFromFile():

Apparently, the printer was able to handle smaller bitmaps but failed once the total size exceeded ~120 lines, 17 bytes each – which is 2040 bytes, and it may be meaningful or just a coincidence. I yet have to figure out whether 2kB is the real maximum limit of bitmap chunk size that the printer accepts.

The printBitmap() (and my file-based variation of it) has the optional LaaT argument – it stands for line-at-a-time and means sending a separate print command for every single line of the bitmap, instead of one command for the whole image. Enabling this flag fixed the printout for me, however using it slows the printing down, so it’s not an optimal solution.

I ended up setting a maximum single chunk of a bitmap to 50 lines, i.e. the 135px high bitmap would be printed in 3 chunks. This is obviously transparent to the user and not visible on the printout. The value of 50 lines is arbitrary, it works for me so far, but given the 2kB hypothesis is right, it would fail for a full-page-width bitmap (384 dots which is 48 bytes, and times 50 it makes 2400 bytes). I’m going to verify it later on.

At last!

Next steps

Having sorted out the basic issues around printing bitmaps, I have the following on my to-do list:

an algorithm to convert actual image files to printer bitmap format (perhaps a .bmp format would be a good starter); once it’s ready, experiment with different images to fix and optimize bitmap printing code

an algorithm to create QR codes in printer bitmap format

Both of the above would enable some real-world use cases for the thermal printer, so stay tuned for more

But I feel like I need to explain the whole thing from the very beginning.

A thermal printer?

So a couple of weeks ago I met with two old friends of mine to take part in the hackaton in Kraków, Poland. The hackaton was actually just an excuse to get together, take a little trip down memory lane, and have some potentially inspiring conversation over drinks. Mind you, we did some coding at the hackaton and even got the prize, but still, the late Saturday beers made the real impact.

That night I learned that there was a guy on the internet that made a thermal printer automatically print tweets he had been mentioned in. It just blew my mind how nicely this matched the definition of hacking, which is, by Wikipedia:

The act of engaging in activities (such as programming or other media) in a spirit of playfulness and exploration

Straight off I wanted to do a similar stuff myself. Not necessarily the Twitter thingy, but the possibilities are infinite. Plus when it comes to tinkering with electronics, in case of a thermal printer, the possible benefits like saving your time or making your life easier are inferior to the coolness of possessing the actual thermal printer.

I mean, do you know anyone that owns a thermal printer for personal use?

What’s on the market

The go-to solution is the Adafruit’s Mini Thermal Receipt Printer, and it’s easily available in Europe too. There’s a library for Arduino and Python provided by Adafruit, as well as some example code. At $50 however (or over €60 where I live) I deemed it a bit too pricey for a toy with no real-life application in mind (yet).

There are obviously plenty of thermal printers on AliExpress, so I went there and found one in a similar shape, hoping that it would come with the same internals and communication protocol too. Sacrificing availability for a lower price, I ordered it and patiently waited for the shipment.

4 weeks later

When I unwrapped the package, I noticed that my printer had a different connector than the Adafruit printer. That wasn’t a big deal though, as the only outlets you need for a thermal printer are power and serial port. You can even skip the printer TX pin if you won’t be checking the paper status. All of these pins were provided on my printer’s connector so I was ready to give it a spin.

First test

I wanted to check if it works with Adafruit C++ library for Arduino. I set up a test circuit with ATMega-328P microcontroller, ran the example code and it worked! Well, sort of.

I could see the paper coming out of the printer, and it had words printed on it. Not all of them though, given what was in the source code. But as you can see from the photo, I powered it using the iPad charger that’s reportedly capable of sourcing up to 2A@5V DC.

The printer, according to datasheet needs around 1.5A at 5-9V DC, so technically it shouldn’t be a problem. However a friend of mine that bought the same printer at the same time powered it with 8V DC and got a flawless printout, so I knew I needed to sort out the power supply.

In fact, the next day when I connected it to the shiny new 2A 9V DC power adapter, the example code for Adafruit Arduino library worked perfectly fine with all the features listed in the example program, such as printing barcodes, QR codes, and bitmaps.

Having confirmed that my printer works with Adafruit library, I went on to get it working with LoPy from Pycom.

MicroPython

As my WiPy is busy running the air quality monitor, and the LoPy4 hasn’t been shipped yet, I managed to borrow two LoPy version 1 boards from a friend – mainly to evaluate LoRa, but also to try out the printer.

Adafruit were generous enough to put up a thermal printer library written in Python, so I based my work on it, and did the following changes to get it running:

removed Python 2.x code since MicroPython is Python 3,

removed writeToStdout(),

replaced Serial with machine.UART,

removed image printing method, because it depended on Python Imaging Library that apparently is not available in MicroPython.

This is what happens when you omit a stop bit.

Still not bad for the first try The next step was to get the UART’s stop bits right, and then it started to look similar to what I saw on the ATMega with Adafruit’s library. I then followed up with some improvements, including:

removing support for pre-2.68 firmware – my printer is 2.69 and I don’t have older models to test with,

disabling calls to wake() and reset() at initialization because they caused printing garbage in the first line printed after initialization,

fixing sleepAfter() and wake() methods that used printer commands different from those specified in the datasheet. It was either a bug in the original library or a feature of older firmware versions. It might as well work just fine with my printer too, but I was in “check that everything is in line with the datasheet” mode as I was constantly getting some unwanted characters printed out here and there;

parametrizing heat dots and heat interval settings – my printer had different settings than those hardcoded by default in the Python library;

there’s an API for getting that info: calling testPage() will print the test page (!) that also contains information such as current temperature, operating voltage, firmware version plus heat dots, heat time and heat interval parameters.

And that’s how I eventually got it right. The underscored indent before “Large” that is visible in the photo below got fixed in the meantime too.

Just so you know, printing with the lib is dead simple. For regular text it boils down to just calling println():

Text decorations like changing size, alignment, font weight or line spacing are all single-line commands too. There’s an example code in the repository printing the text shown in the photos.

Remaining work

I still have to make printing bitmaps work, as my LoPy couldn’t manage to do that due to memory allocation problem. It might be a device limitation (it’s got 512kB of RAM but I guess some of it is already taken by MicroPython runtime) or a bug in the code, or the combination of those two. I’ll give it another try in the next couple of days, maybe I could make a proper use of garbage collector… I’m still learning Python so I could as well be talking bullsh** – I’d appreciate any kind of help or guidance if that hurts your eyes.

Disclaimer

I need to point out again that the printer I tested the library with is not the original Adafruit Mini Thermal Receipt Printer so your mileage may vary with their device. Given that I used Adafruit printer’s datasheet as a reference and achieved satisfactory results, chances are it would work with their printer too.

If you happen to have that printer and would like to give it a spin with a MicroPython-powered microcontroller then I’d be happy to hear about your experience. And if you don’t have the thermal printer yet, I wholeheartedly recommend getting one, as I believe it’s a beginning of a great journey Once again, you can get the MicroPython library from GitHub.