This project is submitted for

Description

IoT Framework with mesh nRF24L01+, STM8L, STM32, mbed-os, and nRF51, Raspberry pi, MQTT Mosquitto, OpenHAB2, json, mongodb, redis, iotdb, websockets, reactjs webapp, d3js. What a better chance to learn the basics and not get stuck with blackboxed tools. Learn C, C++, javascript and Python. Start as a complete beginner with on the shelf MQTT clients, end up hacking a custom RF protocol and an advanced time series database and standard Dashboards.
The T from IoT is met with an easy to make Coin cell sensor tags and an even easier mesh of repeaters, connected to the Raspberry pi. Have fun controlling light with a $3 gesture sensor or show the humidity status on a 20 cents RGB LED.
All SW and HW is opensource on github.

Details

nRF51 Beacon support available !

main code already on github, more description to be provided soon. I used the nRF SDK with the proprietary_rf examples esb_low_power_ptx. The modified code makes the beacon behave as a dash button acting like all other mesh RF nodes.

STM32 IoT Board

This board comes with multiple functions (Gesture and proximity, RGB status, long range repeater and RF to serial)

Cell Coin Sensor Tag

STM8L151F3P6

BME280 : Temperature Pressure Humidity

MX44009 : Light

MRMS211H : Magnet sensors

Usage of modules very simple to solder, can reduce the cost vs on the shelf tags by soldering only the required module

Grafana Interactive dashboard snapshot

There you can experience an interactive dashboard showing one day of measures similar to the screenshot below

Notice the logarithmic display of the light values that show similar curves for different light sensors on different rooms, the passing clouds patterns can be seen in the signal, as well as differentiation between natural and artificial light (sudden change). This is just the beginning as further sensors already deployed bring even the color of the ambient light.

Installing your own InfluxDB and grafana server would get you the same for your daily real time measures, all of this owned by you, locally on your raspberry pi.

Mesh Protocol

Broadcast flood, simple peer to peer with end to end CRC and acknowledge, streaming mode without ack, controllable time to live adjusts to the mesh scale.

MQTT as a sensors interface for time-series databases

MQTT as a Multi-level interface for Actions

Custom RF : Simplicity Vs Complexity

With Bluetooth, you need a bigger SW stack (hundreds of kilobytes, while it is few kilobytes here 2~ 3 KB)

With Bluetooth, you need a more expensive uC or SoC

With Bluetooth, you cannot customize the protocol to your needs

With a custom protocol you can have a simple mesh starting from few functions, easy to understand and optimize by yourself

With a custom protocol you can have a huge amount of sensors, using a specific configuration.

For simple sensors that do not control anything, no door opening, no light switching, security is not relevant as these sensors are beacon like and do not provide more information than what a neighbor could see out of your window.

The internet security is handled by the raspberry pi that is beyond the custom RF gateway, so you can access your data safely from all over the world, for example I am using a vpn router, so the security is even beyond the rapsberry pi server's hacking.

Project Logs

Mesh use cases

before talking too much about mesh concepts, let's see end results what useful things can we get of this ?

Flood versus Routing

This is a Flood mesh type opposed to routing mesh types.

With a Flood mesh, Zero configuration is required, just throw nodes in, every node has its id.

A Peer to peer message just need to know the destination id, the stack handles an end to end acknowledge with retries if necessary.

This type of mesh is router less, which means it get advantages such as

the nodes do not need routing tables to define the shortest path

which is also hard to configure if the nodes move or would have to be identified on startup.

A routing mesh would also require a central server or duplication of management in every node acting as a server.

But there's no magic in here, a Flood Mesh have some compromises to do which might be completely irrelevant in some scenarios.

The mesh Range is as wide as you have nodes and capacity to live through jumps, yes but then why don't we use a huge time to live for an endless range ?

Well the problem is the echo, as the nodes have no routing info, they will keep repeating messages to each other, even back to those they got the messages from !!!

This Echo is born from the flood concept itself, it reaches all the nodes around who keep singing along for one single message until that message dies.

During a time to live group repetition, all the impacted network is busy with that single message. which highly increases collision chances and messages loss.

In the scenario of a home, this compromise is not of much relevance as you barely need a single repetition jump, one Time To Live can get you through a villa, and two through a huge building, this is not fit for a city network as its name says this is a #Home Smart Mesh.

As two nodes listen to an RF message at exactly the same time, they are likely to react with a repetition at exactly the same time as well. To avoid that, every node uses a per id delay configuration to repeat the message with a shift in time.

Leaf versus Repeater Nodes

Leaf Nodes are acting like beacons. They are wireless low power sensors that cannot keep listening, thus the frequent Broadcast for them is strategically a better compromise than a Message to a server. The unpredictable time listening waiting for the acknowledge can allow to re transmit much more frequently with the same power budget. Additionally, these nodes do not need to be configured to know the server Id.

The Mesh Protocol

This Mesh protocol was custom designed to take advantages of the hardware it was designed with. The reason it is different than others is simply that the RF transceiver nRF24L01+ is much cheaper. With up to x10 times cheaper, well you get some side effects such as packet size limitation. With 32 bytes, if your protocol header is big, well then you have no more room for a payload.

The protocol above uses only 8 Bytes at max while fulfilling all conditions of end to end addressing, redundancy checks, acknowledgment and repetitions.

By design, messages do not have time stamps nor sequence ids, which let the multiple repetitions reception be filtered at the destination side. That means that yes, filtering time has to be slightly higher than the Flood echo time, and the sensors broadcast frequency must be slightly lower than the listeners filtering time.

The only complexity we get here is in the Pid that has been compacted to handle both the message type (e.g. Temperature, rgb color request,...) and the message status, if it is a broadcast or in case of peer to peer Request or response.

The source and destination being part of the user software protocol solves the problem of all radio transceivers that have hardware pipes with addresses. Those addresses have to be configured depending on the network and the number of pipes is limited for the nRF24L01 to 5. Well, it is nicer to have more than 5 nodes in your mesh !!!

How is this working ?

Here we push the number of LEDs up to x216, it could go up to x220 but then it would not be aligned any more. The panel offer up to x256 LEDs but the STM8S memory of 1KB reaches its limit here. For a single color I could simply rework the function to lightup an unlimited number of WS2812B, but we would then loose the option of displaying figures or messages.

Every WS2812B consumes up to 60 mA, so for 256 LEDs that's no kidding 15 Amps !!!! Thus the transformer next to it can go up to 20 Amps !! note that this might be dangerous if you use less powerful transformers !! From safety perspective, I'm currently using a manual switch in addition, I would not recommend such an installation without a relay that fully cuts out the current when not in use!!!!

The radio protocol is very simple and provided in the libraries through the WS2812B.c it looks like this :

PID - NodeID - R - G - B - CRC

MQTT Control

For convenience here, I used a ready Android application MQTT Dash which worked out of the box once I used the topic payload format for colors which is the #RRGGBB (e.g #FFFFFF for white). It is enough to program the right topic address (e.g. Nodes/10/RGB), this app is cool as it even provides colors pickers.

By the way, we can see here a TVSet Power as well which is an off the shelf Wemo switch connected through OpenHAB2 and MQTT.

Conclusion

If you need just brighter light for your room, do not use this Panel which is more valuable than that, I plan to shit it to a central messaging notifiers and use it to display text and symbols.

Demo

We can see here how the dashboard looks like, definitely not impressive but the good news is that it is completely hackable !!!

So we can see there bullets going up with the temperature changing color accordingly, note that the color matches the gradient position of the rectangle it's in. The curves show the last 24 h, still planning to add dynamic real time animations, the technology allows but not the time. This page shows an ugly text debug window at the bottom helping me to check all received notifications.

What's behind this ?

first of all, this is web, only web, html5 is helping, javascript is improving and the tools coming with it as well to make it a descent programming language.

I struggled looking for ready javascript libraries to show graphs such as highcharts or amcharts they look nice, but not only you get stuck with licence stuff but you cannot edit it to very customized display, the right choice ended on d3js

D3js

d3js is a wrapper library around the svg standard which just makes selections easier and have a lot of helper functions and examples

The learning curve is long, but I highly recommend it to those who want to understand the web technology and not simply use it.

Websockets

Another major improvement in the web, after struggling for server side events with comets or other alien like mechanisms, the websocket restart from scratch and provide an efficient tool already known and proved, the hack behind it is to start as an http request, and then upgrade to a websocket

Poco

I know javascript is fancy, I know python is cool, sorry to say this but c++ is easier, here I'm not talking about personal preference. Not about writing code only which is obviously faster if you omit declarations, but I'm talking about the effort you spend from when you start writing your code till you get it running as you expect.

That is why I opted for a C++ application for netwroking where Poco marvelously does the trick. I hesitated between boost io and poco, then opted for poco as it is more than a library as it provides even application class, log and other helpers.

Security

Some of you might wonder how would I secure this, with all polemics we read about every mentioning IoT devices being hacked, and I do yes access this remotely.

VPN : yes, the answer is simple, let us not re-invent the wheel, plus I have no time to spending updating every new patch of my authentication web interface. I admit here the solution was expensive, a VPN enabled router, but it's a single time payment for a multipurpose router and the result is independent from any 3rd party (other than simple domain name redirection).

It is important to note here how users can and must own their IoT devices by using tools that do not require third parties services that only work with registrations or even worse requiring active internet connections, here latency and confidentiality have to be taken into account.

Some Javascript

I learned how to split my javascript in more than one file (not trivial, yes, I'm serious), so I isolated the usage of the d3js graphic elements from a simple interface:

Demo

On the down right side of the video is the RGB Led status indicator that turns blue when the bathroom is humid (Humidity = Water = Blue) and green again when it is time to close it not to use much eating energy for example.

In this video, I closed the door to raise the humidity level, then opened the door and the window to evacuate it.

Installation

STM8L wireless sensor node with its BME280 as a humidity sensor

An RF Repeater node that is visible on the down right side of the video which has an WS2812B as an RG led status indicator

A raspberry pi with an RF dongle as depicted in the Gallery HW + SW design

Software : MQTT Ruler

The relevant software to fulfill this action in addition to the already installed HW+SW, is simply one small MQTT client, I wrote it in C++ but it could be in python as well, this client is nothing more than a ruler, it subscribes to an MQTT message (humidity level of the bathroom sensor id), and then produces a color depending on the level and send it again through another MQTT channel where the RF dongle is subscribed to send an RF signal to the right node id with the color to set.

This code extracts the node id from the Topic, converts the message payload from text to float and sends them together to the "publish_humidity_status()" function which is testing the node id, applying the rule and publishing the message.

The function applying the rule which converts the humidity float into a color can be seen below

Demo

Firmware

The 60 x 3 BYTEs info for RGB of every LED are first updated in the STM8 memory, then sent out.

Shading colors

What might look sophisticated for an LED application is a basic operation for a graphics application background

//inputs: the leds section to use defined by which led to start with and which one to end before//inputs: The color of the first led to use and the color of the last led to use//action: updates the leds memory with the first and last led colors and all intermediate leds are interpolated//comment: This function does not send the led data bits on the bus, so that multiple operations can // be applied before sending the whole result together with SendLedsArray();voidrgb_Shade(BYTE LedStart, BYTE LedEnd, RGBColor_t ColorStart, RGBColor_t ColorEnd){
int nbLeds = LedEnd - LedStart;
BYTE LedId = LedStart;
for(int iCount=0;iCount<nbLeds;iCount++)//0-10
{
RGBColor_t Ci = rgb_ColorScale(iCount,nbLeds,ColorStart,ColorEnd);
rgb_SetColors(LedId,Ci);
LedId++;
}
}

Demo

This was the very first step in this project that shows how machines talk to each other in a way humans could perceive :

Broadcast use case

Firmware

WS2812B bitbanging

The challenge here was to make an 16MHz 8bit microcontroller namely the 1$ STM8S bread board talk to a WS2812B where the full fledged ones have to be using tricks and DMAs and timers and complex stuff.

As you might know there Smart RGB Leds are controlled with pulses that generate the 24bits of reg green blue, the problem is that every bit a pulse short or long for 0 or 1, and the Time for the high pulse of a zero as an example is 400 ns, well at 16MHz, every intruction is 62,5 nano second, that leave us with 6 instruction for looping and deciding which bit to bang the next.

The hack here was to jump down to assambler, yes in this world of javascript, if you really want to hack stuff that's the ultimate way.

rgb_sendArray is a C function, that starts by disabling the interrupts after saving the interrupts state to restitute them later on.

of course any interrupt here would corrupt the bit banging, so expect a latency on your interrupts handling.

The trick to have a multi line assembler that looks nice was to add the end line inside the asembler text.

This function can bag as many LEDs as you have RAM, unfortunately with the STM8S I could not compile more than ~ 220 bytes of continuous memory, which is more than enough for series of Leds.

The trick is to start by setting a PIO, which is a macro, as the pin is configurable, I did not manage to have text confurable variable inside the assembler, so that was the trick.

You always have to set a PIO high either it is a zero or a one, then as they have different timings, if it is not a zero then keep some more nops to make the one pulse longer, otherwise jump directly to the reset instruction to make the pulse shorter.

Now the problem was that looping instructions were not fast enough to cover all cases of second half of bit timings, that means that if we have to send the first bit, then loop, test and start sending the second bit, the loop would be so slow that it is not possible to set the new pulse soon enough. The only way to solve this was to unroll the complete 8 bits, yes unroll which means writing the same code 8 times, for bit one, bit two, bit three. I admit that sounds completely stupid from advanced programming perspective, but if anyone else could hold the challenge of banging WS2812B with the STM8S differently, I'd be very interested.

Any way, it's working, it's not very much code consuming, and it is very memory friendly as every bit to send take only one bit out of the memory, so that we save the RGB values in memory first as three consecutive bytes.

Color Flashing

Build Instructions

I'm trying a new experience with online BOM that could facilitate the production and purchase, otherwise the BOM is also available in the zip file, though to facilitate the DIY production, these boards are based on modules.

The MAX44009 module is referenced as GY-49 (both modules are also very easy to find online and not from raw components suppliers)

MRMS211H, STM8L151F3P6 and resistances are raw components.

2

STM8 Firmware

The firmware is located in every corresponding directory close to the Rpi SW in the main IoT_Frameworks repo.

This repo contains a main libs folder in the root that contains the drivers that are shared by all applications.

It is possible to use free compilers, but I opted for a fully integrated solution including debug capabilities with the IAR for STM8. It is free for non commercial and requires registration but fully worth it.

You need an ST-Link_v2 which is available online for very cheap.

I use the ST Visual Develop to configure the option bits, verification and EEPROM programming (that where the nodeds are)

You need to install any driver for your USB to serial adapter (e.g. CP2102)

Note that you can add the include of the IAR compiler to have automatic completion

3

Raspberry Pi SW

I know that the Raspberry pi SW installation is not trivial, there's plenty of google entries for every topic, which I required for all steps I've done. I am also not an advanced Linux user and cannot improvise an install or compilation from source that is not described in another wiki somewhere, so here is simply the install list I performed :

boost (described further in the main github repo)

POCO : Network components for c++ HTTP servers, websockets and others

SAMBA : for a convenient access to the RPi and work on windows environment

Discussions

Become a member

I was wondering how does the 24L01 6 channel limitation applies here. Does that mean I can't have one node talks to more than 5 nodes in the same level? If I have, let's say 50 leaf-nodes (STM8), how do I get all the sensor data back to RaspberryPI with minimal number of repeaters?

Hi @liveej, this protocol is designed for a home, not for wider buildings as in such case, the main shortcoming would be the scalability. For example, it cannot scale from a house with let's say 50 Nodes, to a complete building with 500. It is restricted by the depth of repeaters that exponentially increase the echo. You still can set up let us say 100 Nodes in every room with only one repeater for that room. I'm currently testing with a time to live of one, a single repeater, but a depth of up to two is reasonable.

Have you considered RFM69 instead of the NRF24 ? It should have no problem covering a house without any repeaters.

I have started my own home automation with the NRF (actually another clone, RFM75) but dumped it. I always thought mesh is very sensitive (seen it nice in research under the assumption that the whole place is littered with always on repeater nodes).

Well actually yes, but it's a complex decision. (+) Technically, a lower freq has better walls penetration for sure. (-) modules are kind of more expensive, (-) sub 1 ghz regulations are kind of more complex. (-) more power comes with more consumptions up to 130 mA, the nrf24l01p has also the pa lna version with 100mA up to 1000m, so that's not for the low power battery nodes. (+) the RFM69 has even encription.

I 'liked' your Stockie project. Have you considered using the stm32? stm32+RFM69 might be something.

Actually custom mesh network is very easy, even trivial if you stick with a depth (time to live) of one or two. An always on repeater within reach in a house is a reasonable assumption. Low power batteries repeaters is on my list but that's indeed a whole different story.

The RFM is indeed slightly less efficient (comparing both at 0dB output power). Honestly, I find too few places where AAA are too big to use.

In 'Stockie' i found the NRF to not be a problem - these modules can actually cover a pretty big area (with the central node being a PA/LNA). BUT: with tens of WiFis around, it sometimes takes a lot of attempts to send a packet - when they find a gap. This is fine for the application, not fine for others.

I have at some point earlier decided to switch to ARM and did consider the STM32, among others. Coming from AVR/Xmega atmel familiarity did play a role, but in the end 2 things mattered most: 1. Atmel studio is way more convenient for a hobbyist to us (ST has no interest to provide a free compiler when they are collaborating with the expensive companies like IAR). 2. The low power STM32L are quite expensive.

So I went with the SAMD21. Since the arduino zero now uses it, i am hoping that soon we'll get arduino clones for cheap (I don't care about the arduino part, just the cheap hardware).

The interference is really a low power killer, I prefer to resend the measures more frequently rather than to waist energy on listening, ack waiting and re transmitting.

The micro controller choice is something I re consider from time to time. I started my hobby career with dozens of PICs from Microchip, technically now owning Atmel as well. It's weird to see how ARM is gaining field, so either you use an Atmel or an ST, it is the arm-gcc compiler or the armcc after all. When it comes to IDEs, I lost faith in them as they take part of the fun away, and I believe that if everyone does a good job of providing standard interfaces (compiler, gdb,...), you could keep using your favorite IDE independent of which CPU you chose, PlatformIO is going in that direction.

Thanks for the link, I'll have a scan of it, it is always good to have a Hackaday entry point to help people find such projects.

There are different trade-offs between the 2 modules. I think that the lowest power can be achieved with the NRF, provided the communication distance is small. In my tests, modules close to gateway could have 90% chance of sending a packet, while those far off in my apartment were dropping to even less than 10%. With the nrf all of them stay at more than 90% success rate. I was saying in my blog post that the ideal module would be a NRF with higher power (>10dB) and RSSI, so one can turn the transmission power down when not necessary.