NicTheGeek

I've been working with IT for a long time...

Store your TTN node data in a NoSQL database, then show it as a line graph on a webpage…

This blog post is all about building a Node.js server to retrieve your LoRaWAN device data from TheThingsNetwork (TTN), then store the data in a Mongo NoSQL database. The assumption is that you have built a Node.js app before. There is also a very simple client app that displays your data as a line graph on a webpage.

The Things Network (TTN) is an open-source, secure and scalable solution that has around 9,000 gateways and 90,000 developers around the world. It started in August 2015 and just keeps on growing.

TTN manages every aspect of creating and managing a LoRaWAN network except storing your data (note that TTN does offer a storage solution, but provides storage for 7 days only). Also note that there are several integrations available, including EVRYTHNG, AWS IoT, Cayenne and Tago.

The following example is useful if you want to keep your data in-house. Also, if you are like me, you want to understand and have control of the key components of your solution, even if you decide in the future to go with a 3rd-party.

This code runs as two separate Node.js applications on your laptop/desktop, but of course when moving to production, the collector app would need to run on a server and the viewer app is actually a web server that would also need to run continuously.

In addition to the code described below, you need:Node.js – javascript server-side run-time environment – I used version 10.16.3 LTSMongoDB – NoSQL database, ideal for json-formatted data – I used a free cloud instance, sign up here (no credit card required) https://www.mongodb.com/cloud/atlas – and of course an active LoRaWAN device, registered within an application configured in your own TTN console – https://console.thethingsnetwork.org/

You can find all the source code, together with the slidedeck I presented at TheThingsConference UK 2019, here:

I have split the code into collector and viewer folders. You need to fill out the connection details for your mongodb instance and your TTN application name and access key (token). Use the config.template as a basis, update with your credentials and then place a copy, called config.js, into each of the collector and viewer folders on your laptop/desktop.

First, let’s take a look at collector; this uses a Node.js SDK provided by TTN that listens for each new published record, then inserts a document into your database. You can see the code for this in collector.js – note that I have created a separate file, mongo.js, for mongodb connectivity, as this is common code for both collector and viewer.

A quick diversion to talk about nomenclature – both SQL and NoSQL databases use the term ‘database’ – but a ‘table’ in SQL is equivalent to a ‘collection’ in NoSQL and a ‘record’ in SQL is a ‘document’ in NoSQL. What I really like about NoSQL databases is that the underlying format is JSON.

OK, so now that we have data in our database, let’s view it in a webpage as a line graph. The viewer folder holds a copy of config.js (note that this only needs to hold the credentials for the database) a copy of mongo.js for database connectivity, the application code in viewer.js and a sub-folder called views that holds index.hbs – a simplified html file.

The Node.js app actually runs the minimum form of express framework to host a simple web page. Our web page is served using ‘hbs’ as it’s default view engine (https://www.npmjs.com/package/hbs) and we use handlebar nomenclature {{JSON string}} to pass data arrays. So, take a look at viewer.js and see that we retrieve a number of documents from the database with ‘getArray’ and then push the relevant data into an array in ‘pushData’ to pass to the web page where we render it. We are going to plot the ‘rssi’ and ‘snr’ as seen by the first gateway to respond to your LoRaWAN device.

The file index.hbs contains the minimal html and javascript to display the data as two line graphs, using the Google charts libraries – more information here: https://developers.google.com/chart

So, in summary:TTN provides an excellent platform to move your node data to the cloudTTN enables simple data access with a Node.js SDKMongoDB is a NoSQL db, good for JSON dataNode.js is a simple way to build a headless server app

Now, it’s down to you to do something useful with your LoRaWAN device!

Protected: Implementing CDC for BigQuery using Matillion ETL

k3s – kubernetes for IoT on Arm processors

I have been working with Docker for some time and run several network-linked containers on a ubuntu server, somewhere in Amsterdam. I am running a pilot project, requiring a single container for each function and so just script the build and connection with docker-compose – here is an example of docker-compose.yml for a project that runs nginx, wordpress, mysql and Node.js

The next step would be to scale on-demand and make the solution resilient by spreading the containers across multiple machines. This is what kubernetes is designed to do, and it’s about time I learnt more…

So, to build a simple kubernetes multiple machine cluster, I could have spun up multiple x86 VMs or cloud servers – but i was fascinated by an article from Alex Ellis – https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/ – it introduced me to a stripped down version of kubernetes that can run in a restrained environment and on Arm.

I looked around my home office and found a pair of NanoPi Neo2 from FriendlyArm – 4 x A53 core AllWinner CPU, 512MB RAM, Gigabit Ethernet – and this, rather unusual offering from globalscale, that I bought from a KickStarter promotion – the espressobin that features 3 Gigabit Ethernet ports, SATA and 1 mini PCIe with 1GB RAM and 2 x A53 core Marvell CPU.

To keep everything stable and provide a heatsink of sorts, I mounted the SBCs on a spare piece of ACM Aluminium Composite Sheet

I installed the current version of Armbian, based on Debian Buster, as a stable ARM64 linux distribution. All 3 are running off 16GB micro SD cards. The default install for the espresso configures Ethernet ports 1 and 2 as LAN, 3 as WAN, so plugging cables between it and the NanoPi just works!

Armbian comes with armbian-config that makes it easy to check for updates, change hostname, etc. I added my default ssh key to each SBC from my Mac using the following command:

ssh-copy-id <username>@<SBC_URL>

Following the the blog I mentioned earlier, the k3s master install on the espresso SBC was simply:

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v0.9.1 sh -

Note that I had to install a specific version as the current build failed with an error – something you get used to when pulling the latest version of code from an early stage linux app. This should ask for your sudo password (you’re not installing as root, are you?) and then install a whole raft of essential tech, including TLS certs, Traefik and Helm – look at k3s.io for more detail. Next, copy the K3S_TOKEN to your clipboard using:

sudo cat /var/lib/rancher/k3s/server/node-token

Now, on each NanoPi, create two environment variables, K3S_URL and K3S_TOKEN, then run the same curl script as before. This time, it will install the k3s worker code:

Alex Ellis is the developer behind openfaas and as such, provides a simple example in the blog post that enables us to test the k3s cluster we have just created. So, just follow the section in that blog, entitled ‘Deploy a Microservice’ and you should see something like this

So, that’s it. try k3s with any available Arm-based SBCs you may have to hand – make it easy for yourself by running Armbian on each. I intend to write a subsequent post, showing some of the automated app deployment, scalability and resilience made possible even out to the IoT computing edge of your solution – happy coding!

Yet another linux on Chromebook post…

I have written about crouton that provides chroot installs of popular linux distributions like ubuntu, debian and kali. I followed that up by installing the official linux on Chromebook offering.

Now I just have to tell you about my new favourite solution – chromebrew

The reason I like chromebrew is that it simply extends chromeos to run like any other linux distribution – “…Chromebooks with Chrome OS run a Linux kernel. The only missing pieces to use them as full-featured Linux distro were gcc and make with their dependencies. Well, these pieces aren’t missing anymore…”

The biggest advantage with chromebrew is that it takes so little space, which is a limited quantity on most Chromebooks. It also has a good package manager, crew.

With crew, it is easy to add the apps you need, try ‘crew install nano’. Package installer scripts are written in ruby and support aarch64, arm7l, i686 & x86_64 processor architectures.Check out crew packages

Quick & dirty antenna comparison for LoRaWAN

One problem working with LPWAN is having confidence that the antenna you just bought is effective. From my experience, it is not reasonable to expect similar-looking antennae to be equally efficient, or even tuned for the same frequency band.

I decided to make a quick comparison of 6 antennae I had acquired over time from various suppliers. The method shown below is intended to simply identify the resonant frequency(ies) of each. The idea of the test rig is that the RF bridge enables matched connection of a white noise source, receiver (RTL-SDR) and the antenna under test. The RTL-SDR scans a suitable frequency band and the signal strength is plotted against frequency. The picture shows the 6 samples numbered 1 to 6 from left to right.

The RF bridge and noise source can be bought cheaply from ebay – as of posting, UK item numbers 232943112992 and 202335272932. The 50ohm terminator came from my parts bin, a relic from the days of ethernet over coax (10base2)…

The idea of the test is first to scan the the rig with no antenna connected. The 50ohm terminator ensures a matched circuit. I chose to scan from 50MHz to 1.5GHz in 1MHz steps and stored the signal strength for each step as the baseline column in a Google spreadsheet – please don’t try this in Excel as manipulating so many values in a single spreadsheet will grind to a halt on a Windows PC.

Now each antenna is attached in turn and the scan repeated, giving columns Antenna1 to Antenna6. Next, each frequency step’s baseline value is subtracted from each Antenna’s corresponding signal strength. This results in a lot of data that effectively indicates the resonant frequency(ies) of each antenna – as shown by the overlaid plots.

So, as an indication, it can be seen that Antenna1 (brown trace) is well-tuned for the European LoRaWAN band, whereas Antenna6 (pink trace) is pretty useless.

Note that this test cannot be used alone to determine best antenna, which would require real field trials, monitoring actual RSSI and SNR at a distance from an installed node/antenna. Resonant frequency, for example can be detuned by the node’s enclosure, mounting and/or large metal objects, such as batteries. An effective ground plane will dramatically improve both receive and transmission characteristics.

However, this test does give a quick comparison in the lab and can lead you to reduce the number of antennae used in field trials.

The ISO is less than 100MB and there are versions for both Intel and ARM-based platforms and I set mine up on nothing fancy (a HP Compaq 8200 ultra-slim desktop with Core i3 and 8GB RAM). It took around 30 minutes to have running the current CE version of docker, together with a ubuntu console (more of which, later).

So, the idea is to boot off the ISO, install to hard drive and then access remotely via ssh – so I can hide the ancient HP box under my desk.

First, create a config file containing the public part of an ssh-key, something similar to this, taking care to indent the 2nd line

A word of caution – this didn’t work for me the first couple of times I tried it; the problem was the layout of the .yml file. Rancher must have seen this before, because they provide a yml file validator.
So, pull the .yml file to your local shell with:

wget http://[your-server-IP]:8000/cloud-config.yml

and run

sudo ros config validate -i cloud-config.yml

OK, so run the install command as above, answer yes when asked and Rancher OS will be installed to your hard drive.

Congratulations! You now have the bare minimum of OS on which to run your docker containers.

But wait – Rancher OS can be made even more useful by swapping out the default busybox console:

Technical overview of LoRa, LoRaWAN and The Things Network

Radio-connected devices work within a certain frequency range. The radio spectrum is allocated by government and licensed for specific functions.

For example, broadcast radio currently has a reserved VHF band from 88 to 108 MHz.

Cellular phones have several reserved frequency bands around 900 MHz and 1800 MHz.

The licensees pay for use within these bands. However, there are some specific bands that are made available for low power use, license-free.

2.4GHz is a global band that allows low power unlicensed operation; this is where WiFi and Bluetooth co-exist.

Some license-free bands are allocated differently by each country and LoRa is designed to work in the appropriate band for each country or region. For example, Europe is covered by 863-870MHz, North America, Canada & South America by 902-928MHz. TTN provide details of global frequency plans – https://www.thethingsnetwork.org/docs/lorawan/frequency-plans.html

LoRa radio technology

LoRa is a technology that works well within a crowded radio spectrum.

It uses a technique called chirp spread spectrum that creates a signal which is distinguishable by the receiver from the atmospheric noise and signals created by other devices that use simpler modulation techniques, such as On Off Keying (OOK) and Frequency Shift Keying (FSK).

LoRa also takes advantage of a feature called Spreading Factor (SF) where the spreading factors are the duration of the chirp. LoRa operates with SF7-12. SF7 is the shortest time on air, SF12 the longest. Each step up in spreading factor doubles the time on air to transmit the same amount of data.

With the same bandwidth, longer time on air obviously results in less data transmitted per unit of time.

So, with very little output power, the LoRa signal can be detected at great distance or through infrastructure that attenuates the signal. Typically, this could be 1-3km in an urban environment. The current record distance is just over 700km from a node under a helium balloon at a height of 40km, at an output power of only 25mw. For comparison your car key fob works at 2mw for a couple of metres and your phone at up to 1W to the nearest cellphone mast, at most 30km distant.

The trade-off for LoRa is that low power and long distance is traded for limited data rate. Maximum data rate at SF7 is 6kbits/s, reducing to 300bits/s at SF12. One key figure RF engineers discuss is “link budget”, bigger is better and Spreading Factor affects the number where SF12 gives the best chance to receive data. For more information – https://www.semtech.com/uploads/documents/an1200.22.pdf

There are payload size limitations for LoRa – in EU it’s 230 bytes per transmission at SF7, 59 bytes at SF12. Consider LoRa only for telemetry data, such as location, heading, height, temperature, pressure and humidity values or simple on-off actions.

The LoRaWAN protocol

LoRaWAN is a communication protocol that runs on LoRa hardware.

LoRaWAN operates in a star of stars network with ability for multiple gateways to receive and forward data from any nearby node to The Things Network.

Each multi-channel LoRaWAN gateway can scan 8 channels simultaneously and decode up to 8 data packets at the same time. Several packets using different data rates may be demodulated simultaneously even on the same channel.

LoRaWAN provides several modes of operation – modes A, B & C, where B is allocated a time slot and C devices are permanently powered on and receiving.

Most nodes run as mode A devices where they occasionally wake up to transmit a small amount of data and listen for a short time after transmission for any received data. Working like this, it is possible to construct a node that can transmit an update every 30 minutes and run off 2 ‘C’-size alkaline batteries for more than 5 years.

Each mode A node is free to transmit at any time, with the limitation in Europe that each node should not exceed a 1% duty cycle.

There are predictive formulae available to show that the combination of multi-channel gateways, small data packets, operational modes and Spreading Factors results in the capability to handle thousands of nodes per gateway. See “Understanding the Limits of LoRaWAN (Adelantado, Vilajosana et al, IEEE magazine January 2017)” – https://arxiv.org/pdf/1607.08011.pdf

The Things Network is provided as open-source solution by The Things Industries. Gateways are being made available worldwide by The Things Network community – see https://www.thethingsnetwork.org/community and try it for yourself! it costs less than £200 to setup your own.

The Things Industries also deliver private LoRaWAN solutions with guaranteed SLA, so be assured that you can develop a solution and build it out to a stable product.

Linux on Chromebook just became extremely simple…

My previous post “My new go-to linux dev machine is a Chromebook…” seems now to be obsolete. Google have released a “linux mode” in Chrome version 69 for Chrome OS. This is the project “crostini” that was first made available on the Pixelbook.

There are a couple of gotchas, such as it is only beta so far and is only supported on certain Chromebooks – see here for more information: linux mode capable chromebooks – note it says that Chromebooks like the HP 14 G3 running 32-bit ARM and others using Baytrail x86 CPU devices will never support linux mode.

Luckily, my ASUS Chromebook Flip C101PA is on the list. So, no need to reset and set developer mode, just simply select Settings -> Linux(Beta) and go for lunch – this is definitely more than a coffee install.

You will see a new app on your Chromebook – “Terminal”. The username and password are those of your Google account – so, launching Terminal, I see a prompt of “nicbkw@penguin:~$”. Click (touch & hold) on the Terminal icon in the appfinder and you have the options to uninstall and shutdown Linux(Beta). Clicking on the launched Terminal icon, you will see the option to open new windows, meaning it is possible to run multiple terminal session windows as well as browser and your other chromebook apps.

The distribution is debian stretch and installing apps is as simple as “sudo apt-get install gimp”. The GUI runs natively in the Chrome OS environment and this linux app appears alongside other ordinary Chrome OS/Android apps – see the header image.

The linux virtual machine, together with any data you store while using it persists between sessions, so you can quit apps, logout, etc without worrying about resetting anything. All data is stored in the same encrypted storage as other Chrome OS data.

My new go-to linux dev machine is a Chromebook…

The other day I arrived at work to discover I had left my laptop at home. I hot-desk some days each week in Liverpool in fabulous SensorCity (@SensorCityUK) and was lost without a keyboard and ability to geek. Fortunately, just 5 minutes walk away is the local branch of a well-known UK “previously-loved” chain, so I went for a meander. I returned with an ASUS C101PA ChromeBook for less than half-price. It has a 10.1″ 1200×800 touchscreen with 6-core ARM CPU, 4GB RAM, 16GB eMMC, USB-C charging and a 9-hour battery – what should a geek do???

The answer is to enable Chrome Developer mode and install xenial ubuntu, of course – note that you should backup before enabling as this will factory reset your chromebook and delete all your data. Warning over.

It is possible to run a full linux desktop (something lightweight, like xfce, bearing in mind the free space on the eMMC), but I like working with a command line and you will see that you end up with linux running in one or more Chrome Tabs, while retaining full access to the Chrome/Android side of things – browser, email, 6Music on BBC Radio iPlayer, etc.

STMicro DISCO-L072CZ-LRWAN1 quickstart with mbed-os

Their code has improved a lot this year. Previously, it was a little tricky to work with – there is now a clean install available on github that can be built locally from the command line on Mac, Windows or linux. using the GNU ARM embedded toolchain and mbed-os.

I describe below a simple recipe to use this code with the STMicro DISCO-L072CZ-LRWAN1 evaluation board (pictured). The code can be simply adapted to run with the sx1272 mbed shield on STMicro Nucleo boards and there is also support for the RAKWireless RAK811 node.

I am assuming that you have already registered your device in your application in your TTN console:https://console.thethingsnetwork.org/applications/[your application name]/devices

1. Setup the GNU ARM embedded toolchain and install mbed-cli:

On a Windows 10 PC – just use the installer that you can download from here:mbed-cli installer

The Windows installer for Mbed CLI includes the following components:

Python – Mbed CLI is a Python script, so you need Python to use it. The installer installs version 2.7.13 of Python. It is not compatible with Python 3

Check the version of mbed-cli and that it is installed OK – version should be =>1.8.2mbed-cli --versionCheck the install location of arm-none-eabi-gcc withwhich arm-none-eabi-gccSet path for mbedmbed config -G GCC_ARM_PATH "/usr/local/bin"