Archives

Archive for the 'Backwoods Logger' Category

The ESP8266 has exploded in popularity since its introduction a few years ago. With easy to use Arduino-compatible modules like the Wemos D1 Mini, it’s like having an Arduino that’s 5-10x faster, has 128x more memory, and has built-in WiFi, all for the ridiculously cheap price of $4. It’s pretty amazing stuff, really.

Like everyone else, I began my ESP8266 adventures with a temperature logging application. Using the D1 Mini, an OLED display shield, and an SHT30 temperature/humidity sensor shield, I was able to create a working temperature monitor in about 15 minutes. It shows the current temperature and humidity on the display, and also runs a basic web server for remote viewing of the data. The only hiccup was the heat generated by the ESP8266, which was enough to raise the temperature about 10 degrees F above ambient temperature when the SHT30 shield was plugged directly into the D1 Mini base. Separating the modules with hookup wires solved that problem. The result was something like this:

Logging and Displaying IoT Data

That’s great for a few minutes of work, but how can I make a single page that shows data from many sensors? Or historical data showing the temperature trend over the past 24 hours? Or a more attractive visualization UI than plain text? Or bidirectional control – the ability to switch some equipment at the sensor on/off? This is the realm of IoT data logging and display dashboard solutions, and there are many options.

One approach is to run a local server to do the job, like a Raspberry Pi or an old PC. There are lots of options for software packages you can install on the server, which will log the data and serve up nice-looking pages to visualize it. I didn’t explore this route, because I was too lazy. I also dislike the idea of connecting to a random 192.168.1.8 server address that changes every time my home network reassigns the address through DHCP. I’m sure there’s a way to give it a fixed address and name like “weather-info.local”, but I didn’t look into it.

There exist a dizzying number of cloud-based logging and visualization solutions. My ideal solution would would have a nice-looking historical graph of the data, with an adjustable time scale, like the example below from Weather Underground. And of course it would be easy to integrate into my ESP8266 software, and cheap or free to use.

I’ve yet to find any solution that I love. Here’s a quick run-down of the cloud-based options that I investigated.

Dweet.io and Freeboard.io

Dweet.io is the data logging service, and Freeboard.io is the dashboard display service. Judging from how frequently they’re mentioned in tutorials and instructables, they’re quite popular. You can also install Freeboard on a local server if you prefer. This setup is for data visualization only, and doesn’t allow for control over the sensor device.

The combination of Dweet plus Freeboard is easy to use, and it looks nice. It would be a great solution except for one shortcoming: there’s no real support for historical data. Freeboard can display sparklines, which are a sort of mini-chart where nothing is labeled and the history is retained in the chart widget itself. Close the web page and re-open it, and your history is gone.

There is an option for a Historical Graph Widget in Freeboard, but it requires upgrading to a paid Dweet Pro account for data storage. That’s only $2/month, but as far as I can tell it means you need to pay before you can try out the historical graph to see if it’s any good. From the screenshots I’ve seen of it, the time axis isn’t labeled and the time scale can’t be modified. It also appears designed to graph a fixed range of historical data, like June 10, rather than a window like past 24 hours.

Freeboard can be extended with plugins, and there are several attractive graphing plugins available. However, all of them appear to function like the sparkline, with the history retained in the graph widget instead of coming from the data source. There may be a solution to do what I need with Freeboard, but I couldn’t find it.

Blynk

Blynk combines the data logging and dashboard display into a single service. It also provides support for control over devices, in addition to displaying data received from them. Along with Freeboard, it seems to be one of the most popular IoT visualization solutions. It’s easy to use, and attractive.

My biggest complaint about Blynk is that visualization and control is accomplished entirely through mobile apps. There is no facility for viewing your data in a web page – everything happens on your phone or tablet. For some people that might be fine, but I strongly prefer a regular computer with a larger screen and easier control over a complex UI.

The Blynk historical graph widget does almost exactly what I want, with an adjustable timescale showing anywhere from the past 1 hour to past 3 months. But the graph is tiny, and can’t be zoomed or enlarged. It also lacks a cursor feature to display the exact value from some time in the past – you have to eyeball the graph and estimate it. When the whole point of the temperature logger dashboard is to dig into the historical data, it’s disappointing.

Blynk uses an “energy” metaphor that I find to be confusing and silly. Each type of widget added to your dashboard requires a different amount of energy, which must be paid for with an in-app purchase. I would prefer a more straightforward monthly subscription model, with the cost based on things like the number of dashboards and amount of data stored. Charging 2x the energy for a data field display with units vs one without units has no correlation to the developer’s actual costs, and makes me feel like I’m playing Farmville.

Adafruit.IO

Adafruit.IO is technically still in beta, although it’s open to everyone and has been around for a couple of years. Like Blynk it combines data logging and display, and provides support for both data display and device control. But like Freeboard it’s web-based rather than mobile. Could this be the perfect solution?

From my limited testing it looks promising. The historical graph widget works as expected, and my only gripe is that the time scale must be chosen when the graph is created, so you can make a “past 60 minutes” graph or a “past 24 hours” graph, but not a single graph with an adjustable time scale.

I’m uncertain what the future holds for Adafruit.IO, or whether it’s still being actively developed and maintained. Since the service is completely free, and has been in beta for over two years, it has the feel of a project that was created with great enthusiasm but was left to languish when the developers moved on to something new. Assuming it doesn’t unexpectedly break or get shut down, it looks like a nice solution.

Cayenne

Cayenne is another web-based logger and dashboard like Adafruit.IO. It was recommended in a recent Sparkfun blog post, but I couldn’t find many examples of people using it. Unlike the other services I’ve mentioned, Cayenne appears to be an “app builder” whose goal is to avoid needing to write any code. Instead you use a web GUI to define what type of microcontroller you’re using, what kinds of sensors, and how they’re connected, and then it spits out a complete sketch you can paste into the Arduino IDE. You can probably guess that I’m not too enthusiastic about this approach. Under the hood, the auto-generated code appears to use something very similar to the Blynk API for communicating with the Cayenne server.

I stopped short after reading some Cayenne docs and videos, and never actually tried to integrate it. Maybe I should give it another try, as screenshots of its graphing functions look quite nice. If I can ignore the app builder and auto-generated code, and just use it as a normal API, it might be OK.

A few other alternatives I looked at only very briefly:

Thingspeak – Somehow related to MATLAB, and geared towards doing extensive analysis of the collected data. Might be OK? Most of the references I found were several years old, and I have the impression it’s been eclipsed by some of the newer alternatives.

Emoncms – Looks somewhat promising, but I think it’s only for local server installs and doesn’t have a cloud service.

I finally had to replace the CR2032 battery in my Backwoods Logger Classic yesterday. The nominal 3.0V voltage had fallen to about 2.7V, and while the software appeared to be functioning, the LCD display no longer worked reliably. The battery was new in June 2011 when I built the first Logger prototype, so that’s two years of battery life while doing continuous logging the whole time! In a post back then, I measured the average current draw at around 20 uA and estimated a battery life of “more than a year”, which proved to be pretty accurate.

The Logger Mini is more of a battery hog due to its OLED display – its AAA battery will last for several months of continuous logging. So the Classic is still king when it comes to long-term data collection.

The Backwoods Logger is a programmable graphing altimeter / thermometer, originally developed here at BMOW and now an open hardware project. Interested in the Logger, but don’t want to build it yourself? Assembled Backwoods Logger Mini units, BLsync adapters, and blank Logger Mini PCBs are now available for purchase! Get one for yourself, or as a gift for your favorite outdoorsy nerd. I brought the first prototype with me on the John Muir Trail this past summer, and it was awesome.

What’s it all about, you ask? The Backwoods Logger is a programmable device for measuring and graphing temperature, air pressure, and altitude. It’s designed for hikers, backpackers, climbers, skiers, trail runners, cyclists, kayakers, snowmobilers, horseback riders, and other outdoorsy people interested in environmental data logging over timescales from an hour to a few weeks. The Logger Mini’s features are:

Graphs of temperature, pressure, and altitude over time

Three graph time scales: past 2 hours, past 10 hours, past 2.5 days

Current rate of ascent/descent

Estimated time of arrival at a user-defined altitude

Weather forecast

Station pressure and pressure at sea level

Snapshot feature – make a permanent record of date, time, altitude, temperature, and pressure at important waypoints

Current date and time display

Imperial or metric units option

Battery voltage indicator

Sound on/off control

3 to 6 month battery life, depending on usage

Temperature measurements in 0.5 degree steps, from -10F to 117.5F (-23C to 48C)

Air pressure measurements in 0.01 in-Hg steps, from 5.9 to 36.12 in-Hg (170 to 1250 millibars)

Altitude (calculated from air pressure) measurements in 2 ft steps, from -1384 ft to 14999 ft (-300 to 4500 meters)

I’ve spent a lot of time getting very familiar with my soldering tools over the past week so that I could bring you these. My hope is that by seeding the community with some pre-assembled Loggers, it will kick-off some firmware hacks and hardware improvements, and take the Backwoods Logger project in exciting new directions.

I’ve also built some blsync adapter boards. Blsync is optional and isn’t required for using the Logger, but it’s a handy tool if you want to do detailed analysis of your graph data. Using the blsync adapter along with an FTDI USB-to-serial converter, you can download graph data from the Logger to your PC, and analyze it using Excel or other tools. The adapter plugs into the ISP connector on the rear of the Backwoods Logger, as shown here.

You can also build your own blsync adapter using the schematic on the project web site, if you prefer.

The price for the Logger Mini is $59, and the BLsync adapter board is $6. Blank Logger Mini PCBs are $3. Shipping in the United States by US Postal Service priority mail is a flat $5.20. If you prefer another shipping method or need shipping outside the US, that can be arranged too.

Please email me if you want to purchase a Logger Mini or other parts. I also have one “factory second” Logger Mini, sold for half price at $29, with some dead pixels in the corner of the screen but otherwise working normally. More details available upon request.

Have you always wondered what would happen to a Backwoods Logger Mini if it were crushed under your own body? No, neither have I, but today I found out anyway. I took out one of the newly-assembled Mini prototypes for a trail run, stored securely in a plastic case in my hip pocket. I wish I could say I was chased down a cliff by a mountain lion or something equally exciting, but the truth is that I tripped on a sidewalk crack before I even made it to the trail. I was running downhill and moving pretty fast, so I went skidding and bumping down the sidewalk with pieces of my hands, knees, elbow, and hip left behind on the concrete. As I hobbled back home, I heard some ominous rattling noises in my pocket. Not good…

Further examination releaved the sad truth: the Mini took a direct hit when I fell, with all my body weight coming down on it, crushing it between my hip bone and the concrete. The plastic case was completely destroyed and smashed to pieces. The OLED glass was crushed, and part of the ribbon connector ripped off. The NEXT button was flattened and the spring mechanism killed. On the back of the Mini, the header pins were bent nearly 90 degrees over, the negative battery terminal was ripped straight off the board, and a bit of wood got stuck in the RTC crystal.

No, it does not still work.

I’m upset at having lost a prototype, since they take considerable time to assemble and the parts aren’t cheap. At least this makes a more interesting story than losing a prototype to a soldering error!

My newest design is a PC downloader utility called blsync, created as part of the Backwoods Loggerproject. BLsync is a command line program with a corresponding hardware adapter, and it enables snapshot lists and temperature / altitude / pressure graphs to be transferred from the Logger to a PC. Above is a quick graph I put together in Excel from blsync data, showing temperature and air pressure in my home over a 40 hour period. If I’d been hiking recently, I’d show a nice altitude over time graph too.

The Backwoods Logger is a programmable graphing altimeter / thermometer, developed as an open hardware project. It’s designed for backpackers, runners, environmental scientists, and other people interested in environmental data logging over timescales from an hour to a few weeks. The Logger design is based on an Atmel ATmega328 microcontroller and Bosch BMP085 pressure / temperature sensor. You can build your own Logger following the published instructions, or purchase a pre-assembled Logger prototype. For additional details visit the project home page, or join the public discussion mailing list.

Bit Bang Serial

The Backwoods Logger wasn’t designed with a PC download function in mind, so retrofitting one after the fact was a challenge. The only external connector on the Logger is the 6-pin ISP port, used for programming the microcontroller firmware. After programming is complete, the pins MOSI, MISO, and SCK can be used as general-purpose I/Os or as an SPI interface. (The other three pins are VCC, GND, and RESET.) So what’s the simplest way to build a PC communication interface using those three pins?

I’m not aware of any straightforward way to build a PC interface using SPI, so my solution was to design a serial interface. Unfortunately the built-in USART that’s normally used for ATmega serial communication is hard-wired to use specific pins, and those pins aren’t connected to the ISP port. Instead of using the built-in USART, then, I wrote a pure-software serial port driver that works over the MOSI and MISO pins. It manually performs all the start bit and bit-to-bit timing functions necessary for a bidirectional serial connection. In order to reliably get the right serial baud rate, I also added some code to calibrate the internal RC oscillator using the external 32768 Hz crystal as a reference. It was all a little painful, but it works.

The Backwoods Logger comes in two versions that run at different clock speeds, which complicated the software serial driver a bit. The Logger Mini runs at 8 MHz, which provides 208 clock cycles between each bit at 38400 bps. That’s a fairly healthy amount of time to react to each bit and decide what to do with it. However, the Logger Classic runs at 1 MHz to minimize its power consumption, which provides only 26 clock cycles between bits. That’s not very much time for processing, and it means I wasn’t able to implement simultaneous two-way serial communication. At any given moment the Logger can either be receiving or sending serial data, but not both. There also needs to be a delay of about 200 microseconds between received bytes, in order to provide the Logger with sufficient time to process a byte before the next one arrives. There’s no receive buffer, so if the PC sends bytes with too short a byte-to-byte delay, data will be lost. The blsync program inserts the necessary delays when sending command bytes to the Logger. Since most of the data flow is in the other direction, this doesn’t impose any noticeable speed penalty. The complete set of graphs can be transferred from the Logger in about 300 milliseconds.

Physically, the blsync adapter is just a passive converter from the 3 x 2 ISP header to a 1 x 6 serial header, with RX connected to MOSI and TX connected to MISO. A MOSFET is used to perform level conversion (thanks Erik!), so either a 5V or 3.3V USB-Serial device can be used. If you know you’ll always be using a 3.3V USB-Serial device, you can omit the MOSFET and connect TX to MISO directly. The 1 x 6 header is designed to work with common USB-Serial devices such as the Adafruit FTDI Friend or Sparkfun FTDI Cable 3.3V.

I also designed a simple blsync adapter board, for a more rugged and permanent solution. It adds a pair of LEDs to indicate Logger and PC activity, but is otherwise the same as the circuit shown above.

Using BLsync

The blsync utility program can retrieve the Logger’s firmware version number, temperature / altitude / pressure graphs, and snapshot list. Data can be saved in CSV format for importing into Excel, or as raw binary.

The blsync utility is configured using command line options:

Usage: blsync -p port [-b speed] [-v] [-c] [-r] [-g filename] [-s filename]
-p port Port to use for Logger communication, such as COM1.
-b speed Bit rate for communication. Default is 38400.
-v Display the Logger firmware version number.
-c Save files in CSV format. This is the default.
-r Save files in raw binary format instead of CSV.
-g filename Sync the graph data, and save it to the named file.
-s filename Sync the snapshot data, and save it to the named file.

For example, to sync the temperature, altitude, and pressure graphs from the Logger connected to COM3, and save them in CSV format to the file graphs.csv, the command line would be:

blsync.exe -p COM3 -c -g graphs.csv

By importing the CSV file into your favorite spreadsheet or other analysis program, you can perform whatever analysis you desire on the collected Logger data. A hiker might create a graph showing his friends how high he climbed or how cold it was. A conservationist might use several Loggers to collect data for a study of temperature and pressure at multiple locations within a habitat, and how they relate to observed species observations. Through hardware and software extensions, humidity or other custom sensor data could be collected as well. Because the Backwoods Logger is an open source, open hardware project, the sky’s the limit on what data might be collected and what interesting things might be done with it.

Sometimes the simplest things give me the most trouble. I’ve been working on a downloader cable adapter for the Backwoods Logger, with the goal of supporting both 5V and 3.3V FTDI cables. Because the Backwoods Logger is a 3.3V design, the incoming TXD (transmit data) signal from a 5V cable needs to be lowered to 3.3V for safe operation. However, the incoming TXD signal from a 3.3V cable should be passed through unmodified. Outgoing signals from the Logger require no conversion, because a 3.3V output is still a valid logic “high” for a 5V system. I need a level converter for a single input, that operates correctly with both 5V and 3.3V inputs with no configuration or jumper settings.

Level Converter Chip

One solution is to use a 3.3V chip with 5V tolerant inputs, like a 74LVC244. That would work, but I’d prefer something simpler and smaller if possible, since I only have a single input to convert.

Clamp Diode

A second solution is to use a series resistor and a clamp diode, like this (image from daycounter.com):

That prevents the voltage at the 3.3V Backwoods Logger input from going more than a diode drop above the 3.3V supply. With a standard silicon diode’s drop of 0.6V, that clamps the voltage to 3.9V. For the ATmega328, that’s not safe: its maximum rated voltage on any input is just 0.5V about VCC. A germanium diode has a drop of 0.2 to 0.3V, so that would work, but it’s not a part that many people typically have handy in their parts bin.

This solution also has the drawbacks of consuming current from the 5V output, and dumping current into the 3.3V supply, raising the supply voltage. The FTDI outputs have a maximum output current of 24 mA. Assuming a germanium diode with a 0.2V drop, that means R1 needs to be at least 62.5 Ohms. Frankly I’m not sure how to quantify the risk of dumping current into the power supply. In the case of the Logger Classic with its tiny CR2032 battery, dumping 24 mA into the battery in the wrong direction definitely doesn’t sound good.

Zener Diode

The approach that appealed to me most was to use a series resistor and a Zener diode connected to ground, like this (image from daycounter.com):

The Zener has a known reverse-bias breakdown voltage called the Zener voltage. Raise the voltage above the Zener voltage, and the diode conducts. The series resistor produces a voltage drop, so that the voltage at the Backwoods Logger input never rises above the Zener voltage. You can get 3.0V or 3.3V Zeners (or lots of other values too).

So I ran out and bought some Zeners, and built this circuit, and it didn’t work at all how I’d expected it to. I used a 3.0V Zener, and a 100 Ohm series resistor, to limit the current drawn from the FTDI cable to 20 mA. When I connected a 5V dummy output, I got 2.91V at the Logger input. That seemed odd, it was off by 0.09V, but it was still close enough. Then I connected a 2.95V dummy input (the actual voltage from my crummy “3.3V” breadboard regulator), and I got 2.4V at the Logger input. Huh? That’s not going to work. I had expected that for any voltage below 3.0V the Zener would do nothing, and for anything above 3.0V it would clamp it to 3.0V, but that’s clearly not what happened.

What went wrong? Truthfully, I’m not exactly sure. The datasheets talk about a minimum current necessary to get the Zener effect, but I’m not sure that applies here. I can’t safely increase the current further anyway without damaging the FTDI cable. But would more current even solve this problem? It makes sense that the Zener wouldn’t really turn on instantaneously at 3.0V, but rather would begin to conduct more and more as the voltage approaches 3.0V. With a voltage of 2.95V, the Zener would already be partly conducting, pulling the voltage seen at the Logger input below 2.95V. But how much below? How can this be quantified?

One thing in particular bugs me about all the Zener diode datasheets: every datasheet lists values for standard measurements called Izt, Rzt, Izk, Rzk, Rz, and a few others. These are standard measures from some hypothetical standard Zener graph, but the datasheets never actually show this graph, and I’ve never been able to find one anywhere. I know “k” is for “knee” and I believe “t” is for “test”, but what I really need is an actual IV curve for a Zener with these values labeled. Then I think I’d understand this better.

Further Complications

Just to make things more interesting, there’s one more constraint to consider. The Logger Classic uses an unregulated battery as its supply. It can work just fine at battery voltages of 2.8V, and probably down to 2.5V or even lower. In order to stay within the VCC + 0.5V margin of the ATmega328P, the input voltage must not go more than half a volt above the potentially fading battery voltage. A standard 3.3V input when the battery is 2.7V would actually damage the ATmega. That’s why I chose to use a 3.0V Zener rather than a 3.3V one. That should be safe down to a battery voltage of 2.5V, below which I could configure the ATmega’s brownout detector to engage.

The Way Forward

I’m going to sleep on this, and see if anything brilliant comes to me. If anyone else has a suggestion, please reply in the comments. Assuming I can’t find a way to make the Zener work while still meeting the other constraints, then I’ll probably cave in and use a level converter chip. Without really understanding the implications of current flowing into the supply battery under the clamp diode method, I wouldn’t feel comfortable relying on that approach.