After building, on the previous posts, the Node-Red based backend to support E2EE (End to End Encryption) so we can log data into a central server/database, from our devices securely, without using HTTPS, we need now to build the firmware for the ESP8266 that allows it to call our E2EE backend.

The firmware for the ESP8266 must gather the data that it wants to send, get or generate the current sequence number for the data (to avoid replay attacks), encrypt the data and send it to the backend.
On the backend we are using the Java script library for cryptographic functions Crypto-js, and specifically we are encrypting data with the encryption algorithm AES. So all we need is to encrypt our data with AES on the ESP8266, send it to the Node-Red Crypto-js backend, decrypt it and store it, easy right?

Not quite, let’s see why:

Crypto-js and AES:
We can see that on my Node-Red function code and testing programs I’m using something similar to the following code example:

The code variable AESKey the way it is used on the above example encrypt and decrypt functions isn’t really a key but a passphrase from where the real key and an initialization vector or salt value is generated (I’m using the names interchangeably, but they are not exactly the same thing except they are public viewable data that must change over time/calls).
The use for the generated key is self explanatory, but the initialization vector (IV) or salt value is used to allow that the encrypted data output to be not the same for the same initial message. While the key is kept constant and secret to both parties, the IV/salt changes between calls, which means that the above code, when run multiple times, will never produce the same output for the same initial message.

Still referring to the above code, the algorithm that generates the key from the passphrase is the PBKDF2 algorithm. More info at Crypto-js documentation. At the end of the encryption the output is a OpenSSL salted format that means that the output begins by the signature id: Salted_, followed by an eight byte salt value, and after the salt, the encrypted data.

So if we want use the API has above on the node-js/crypto-js side, we need to implement on the ESP8266 side both the AES and PBKDF2 algorithms.

I decided not to do that, first because finding a C/C++ implementation of the PBKDF2 algorithm that could be portable and worked on the ESP822 proved difficult, and second the work for porting it to the ESP8266 won’t be needed if I use a KEY/IV directly, and so I decided to use the more standard way of providing an AES key and an initialization vector for encrypting and decrypting data.

In the case of Node-JS and Crypto-JS when using an explicit key and IV the code looks like:

Now, with above code, where the IV is always initialized to the same value, in this case ‘0000000000000000’, we can see when running the above code several times that the output is always the same since the IV is kept constant. Also the encrypted output is now just the raw encrypted data and not the Openssl format.

So to make the above code secure we must randomize the IV value for producing an output that is always different, even from several program runs when encrypting the same source data.

As a final note, if we count the number of HEX characters on the Key string, we find out that they are 16 bytes, which gives a total of 128 key bits. So the above example is using AES128 encryption, and with default Crypto-js block mode and padding algorithms which are CBC (Chain block mode) and pkcs7.

Interfacing Crypto-js and the ESP8266:
Since we are using AES for encrypting data and decrypting data, we need first to have an AES library for the ESP8266. The AES library that I’m using is this one Spaniakos AES library for Arduino and RPi. This library uses AES128, CBC and pkcs7 padding, so it ticks all boxes for compatibility with Crypto-js…

I just added the code from the above library to my Sming project and also added this Base64 library so that I can encode to and from Base64.

The only remaining issue was to securely generate a truly random initialization vector. And while at first I’ve used some available libraries to generate pseudo-random numbers to real random numbers, I’ve found out that the ESP8266 seems to have already a random number generator that is undocumented: Random number generator

If we are deploying several IoT devices, above a certain number it becomes a hard task to know which ones are alive and working, their location, and since they may have different functions/purposes, to keep track of their configuration. This last problem is exacerbated if we need to keep and maintain different code bases and firmwares for each device.

Device provisioning and some cleaver firmware code, can help to bring these issues under control. Device provisioning is nothing more than a fancy name for a central server that holds information for each device, it’s configuration and status and even might allow some form of monitoring. In advanced usage it might let an operator to push firmware or configuration updates to certain devices, without the need to physical get them.

So this post is about my simple device provisioning that allows to keep track of my ESP8266 devices, running the Sming firmware, and control, in this initial stage, some of it’s configuration.

The solution is based in three components:

NodeJs based REST server that holds information in a SQLite database

An Angular.Js based web application to managed the device configuration

Sming based firmware for the ESP8266

The REST server
The REST server is based in javascript running on a NodeJs server, which uses Express and Body-Parser to create a simple and quick REST server. The REST protocol uses the HTTP operations GET, POST and PUT to define operations like get data, create data or update data. Due to the Express modules, we can build a simple interface to a database that translates REST operations to database operations. The Body-Parser module allows to communicate purely using JSON, and so takes care of all the details of coding and decoding JSON data communicated over HTTP.
Finally there is one important thing, at least for the browser front-end that is the CORS protection. CORS, Cross Origin Resource Sharing is a security feature that browsers use to not connect to anything else other than the originating domain of the page that are showing. To allow our REST server to be called by a running application on a browser (in our case, our Angular.Js frontend), the server must explicitly allow this. So we also need a CORS module for our REST server.
Finally I’ve chosen a SQLITE database due to it’s simpler configuration. No need to setup servers…
To allow our REST server to run we need to install locally to our application the folowing modules: npm install express node-sqlite3 body-parser cors

The function db.all returns an empty array if no results, or an array with row set. The row set is returned directly as a JSON array object to the calling application/device.

For the last URL, the implemented verbs are GET, to get the latest configuration and update the lastseen field, POST to register the device and/or returning the configuration, and PUT to update device information (used by the front end):

The above REST verbs, namely the POST and PUT need a request body, to pass data from the source to the database.

In case of the POST verb the, body should be the following JSON: { “ipaddr”:”x.x.x.x”, “ssid”:”wifissid” }. For example:

We save the above code on a file, let it be restserver.js for example, and start our server with node restserver.js.

On the front end then we can have now a list of devices that are registered:

And selecting the device we can change the human readable name, and add configuration properties:

The front end, based on Angular Js isn’t completly ready, since it’s main purpose is to graph data from Sparkfuns Phant server.

Sming ESP8266 Code

To make this complete, we only need to start coding the ESP8266. I’ve abandoned NodeMcu and Lua language, due to several factors. Sming is a great firmware and already has support for a lot of devices and also includes a JSON library.

And that’s it. After startup, the device after connecting to WiFi, makes a request to the provisioning server. If the request is from a new device it registers, if not it receives it’s configuration data. From this point we can do anything, namely pooling periodically the provisioning server to check for new configuration (by comparing the cfgsn hold by the device with the received one).