Mongoose OS and the ESP8266

Introduction

The ESP82661 is a cheap hardware platform for WiFi-enabled devices: you can put something on the internet for well under a fiver2.

However, we need to consider the software too. The default ESP8266 firmware makes the device into something like a WiFi modem: it connects to the main processor over a serial link, and accepts AT commands3. This isn’t ideal, because you’ll need a second processor in the box to handle the application code, even though in many cases the ESP8266 can do it all.

Mongoose OS4 is one alternative. It provides a replacement firmware for the ESP8266 which includes a psuedo-Javascript interpreter and webserver. So, a typical ESP8266 Mongoose OS project contains:

The OS, by which I mean both the traditional OS which handles networking and the like, plus extentions to the Javascript engine to handle hardware. Like other operating systems, it includes a number of daemons, some open to the network.

‘User-space’ files which include can include Javascript files containing ‘application logic’.

The code above downloads a blob containing the OS and a file-system from https://mongoose-os.com/downloads/esp8266.zip, then flashes it to the board. At this point a LED should start flashing.

Some gotchas:

If you have a directory called esp8266, mos will try to find a firmware blob locally instead.

If you have multiple plausible serial devices, you might need to tell mos which one to use with the --port option.

If your LED is on a different GPIO, you’ll need to edit the code.

WiFi

At this point, you might wish to configure the WiFi connection:

$ ./mos wifi SSID PASSWORD

In practice, it is often easier to do this from the browser-based IDE.

Mac Flashing

On my Mac I found it impossible to flash boards which use the CH340 USB-serial bridge. Once flashed, everything worked well though. Rather than explore this particular rabbit hole, I flashed these boards from a Linux box instead.

The integrated IDE

Mongoose contains a full web-based UI, which allows you to all the things you can do from the command line and more. For example, you can flash the ESP8266 or configure the WiFi with the IDE instead of at the command line. To invoke the IDE:

$ ./mos

At this point your web-browser should open something like http://127.0.0.1:1992/#files, which of course is a web server embedded into mos.

You should see a pretty UI which lets you explore the device. For example, you can browse files on the ESP8266 by clicking on the ‘Device Files’ link on the left-hand-side.

init.js is a key file: it’s essentially what gets run at boot, and so by looking at it, we can tell what the device is going to do. You can either use the in-browser file manager, or the command line:

So we can see that besides flashing a LED, the NodeMCU has also been configured to make a MQTT10 request when a push-button is pressed.

The MQTT server is configured in the confN.json files: these form a crude overlay database when things defined in e.g. conf9.json override things in conf0.json.

You can also configure MQTT from the ‘Device Config’ section of the UI, or by using mos config-set.

The filesystem

Having got to this stage, it is easy to edit files on the ESP8266. You can either get and put files with the mos tool, or just edit them live in the UI. Either way, you’ll probably have to reboot the ESP8266 for the changes to take effect.

For example, having flashed the default firmware, you can change the LED’s period by just editing the 1000ms interval between timer events. You can then save this new init.js, reboot, and see the change. There is no need to reflash the ESP8266 from scratch. Instead when you save the file, you make an RPC call to the Mongoose OS on the ESP8266, which puts the data into the filesystem.

So, one approach to building a new Mongoose OS system from scratch is to:

Flash some generic firmware.

Upload whichever files you want on the device.

Set config variables as needed: it seems better to use mos for this rather than writing a confX.json file directly.

It will probably become obvious quite quickly that Mongoose has a flat filesystem i.e. there are no directories.

Network access

It’s probably worth emphasizing that once you’ve flashed the basic firmware, all the subsequent interactions need only to exchange blobs of data. So, although it might be convenient to keep using the USB-serial bridge, once the ESP8266 is on the network, you can do most of this remotely too.

You can see some examples of this in the RPC documentation11 on the Mongoose website.

This flexibility obviously has some security implications: if you do something over the network without any kind of access control, so can anyone else!

Building an application

If you step back a bit, it is clear that all this configuration malarky just changes bits in the filesystem, so we could avoid doing the configuration step if we flashed a correctly configured image.

Mongoose makes this easy, and probably even encourages such a development model. In Mongoose jargon, a customized blob like this is called an ‘app’, and the process for building them is well documented12.

Abstracting away from details, executing mos build, takes the app recipe specified in mos.yml, and turns it into a blob suitable for flashing to the ESP8266.

The toochain for building this is supplied as Docker container. You can either run it in the cloud or locally.

There are advantages to this approach over flashing random firmware then tweaking it:

Conceptually it is nice to have a single blob containing all the code and data.

You can configure the kernel as you wish, in particular disabling daemons you don’t want to both reduce the attack surface and save disk space.

Given that you have to flash the device anyway, it is quicker to flash it with the correct data than to flash a generic image and then update it.

However, there is a penalty to pay too. It takes a while to build the image, and you have to flash the whole device. So, if you’re just changing a few things in e.g. init.js, it is much faster to edit the files on the ESP8266.

As mentioned above, flashing CH340 boards didn’t work from my Mac which made the process even slower for me. YMMV13!

Javascript

Although Mongoose talks about writing code in JavaScript, this isn’t quite true. You actually write code in mJS14, a limited subset of JavaScript.

I think it would be better for all concerned if the Mongoose documentation made this clearer. I wasted ages trying to get some code to work, only to discover that mJS does not support closures. The mJS GitHub repo does include a list of restrictions15, but I didn’t know that at the time.

The lack of closures mean that many callbacks take a void * pointer to userdata16.

I think the main difference from calling mos is that I made RSA certificates, but it might be better to use ECDSA instead. As this forum post22 explains, ECDSA will be a lot faster if you have a ATECC508A crypto-chip. On the other hand, I don’t have such a chip!

It is worth pointing out that connecting to AWS IoT does take ages: about half-a-minute in my experience. This isn’t Amazon’s fault: the ESP8266 is slow23.

Whilst clever and potentially interesting, I note that connecting a Mongoose device to AWS IoT does not mean you’re opening a connection purely for data.

STM32

Another headline Mongoose feature is support for other chips. Although I’ve mentioned ESP8266 a lot above, I’d hoped I could substitute ESP32 and STM32 without a problem.

However, whilst Mongoose stand by their ESP32 support, the STM32 stuff is presently ‘really flaky’25.

Conclusions

Overall I’ve been both delighted and disappointed by Mongoose OS.

On the plus side, it makes it very easy to build a certain class of projects based around the ESP8266. You can get source code from GitHub26, you don’t have to go far to get reasonable documentation27, and the forums28 are great.

On the other hand I think my expectations were set rather too high:

You can’t program in JavaScript but instead a subset of it.

The STM32 support doesn’t work.

Also I think the ESP8266 just isn’t fast enough to let you write even slow device handlers in mJS. For example, I wanted to connect a rotary encoder, but it was easy to turn it too fast to track.

In practice I think anything which goes up to the JavaScript layer takes a randomish time of about a few ms. For example, I hooked a scope to the blinky example and found that although the average flashing period was pretty accurate the standard deviation was about 1ms, and after a thousand iterations I had about 4ms spread between the minimum and maxiumum periods. I think that means that if you want to handle frequencies higher than a few Hertz, you’ll need to use C.

Writing in C isn’t a problem, but I’m not sure I want to write lots of it against the Mongoose APIs or in the Mongoose environment. I think I’d prefer to work in a more traditional setting, but that might just be ignorance.

There’s also a subjective, aesthetic, issue. I feel there’s quite a lot of magic built into mos so you either have to take things on trust, or spend time and thought working around mos.

Take the AWS credential issues: I wasn’t particularly keen to divulge my AWS secrets to Mongoose, nor am I particularly keen to outsource certificate creation to Mongoose. I would have been much happier if mos had generated a script which I could eyeball before executing.

I think this is a particularly serious issue with anything security related, so I see a dark side to many of the clever remote management features.

My tentative conclusion is that Mongoose is a great way to build things which:

are based on the ESP8266;

live on safe networks;

contain only hardware which is supported by existing APIs.

Happily, that’s a reasonably interesting subset! It’s probably also fair to say that with more experience, I expect I’d be happy to enlarge this domain.