Thoughts about computing and project management

In a previous post I had posted code that transmitted a message through an infrared link between two Arduinos. The main issue with this code is that if the infrared beam is intermittently blocked during the transmission, e.g. by waving your fingers in front of the IR LEDs, bits will be dropped and the message received won’t match the message sent. In this post I describe a simple protocol that provides error detection of the received message. In my weather station application, the communications is only one-way and so there is no mechanism for the receiver to re-request a garbled message; the message is sent periodically with updated temperature and barometric pressure data.

The message is put in a simple “envelope”. The format of the packet sent is as follows:

<STX><message buffer><ETX><chksum>

where:

<STX> is the “start of transmission” marker and is the character 0x02.

<message buffer> is the text message we want to send.

<ETX> is the “end of transmission” marker and is the character 0x03.

<chksum> is a single byte check-sum of all of the characters in the <message buffer>. You can see in the code that this check-sum is calculated by looping through the message buffer and adding the values of the individual characters.

A check-sum is not going to completely guarantee error detection but it is simpler to implement than a Cyclic Redundancy Check (CRC) check, and is sufficient for my application.

Below is a sample program that transmits “Hello” over the IR link. The function sendMessagePacketconstructs the packet:

The function receiveMessagereads from the serial port, discarding all characters until it receives a STX. Once the STX is received it reads characters, adding them to the message buffer and calculating the check-sum. If an ETX is received, it assumes the next character is the check-sum. Once the check-sum is received, it compares it with the calculated check-sum; if the two check-sums do not match, the message is discarded.

I am using these functions to send text data only and so I am not expecting the bytes that represent STX (0x02) and ETX (0x03) to be in the message buffer. Note that once the message is received, it is null-terminated.

Note on compiling the code above: WordPress won’t render the null character in the HTML code listing above and so you need to replace NULL in the code listing with the <single quote><backslash>0<single quote>. I’ll see if I can fix the rendering issue.

This is a first in a series of posts where I describe how I built an Arduino-based weather station, positioned outside my house, that transmits temperature and barometric pressure data through a window using an IR link to an Arduino-based receiver…

In a previous posting, I had described my experiments with IR LEDs and an IR photo-transistor. I finally purchased a miniaturized receiver for infrared remote control systems, the Vishay TSOP38238. This device contains a photo detector and a preamplifier in one miniaturized package, all for less than 2 dollars. It handles a supply voltage of 2.5 V to 5.0 V and therefore can be powered off the Arduino’s 5.0 V power output. The data sheet for this device can be found here: tsop382

The main advantage of this IR receiver, besides the preamplifier, is that it ignores IR light unless it is modulated (pulsed) at a specific frequency. The last two digits of the part number indicate this carrier frequency in kilohertz and so the TSOP38238 works with a 38 kHz carrier frequency.

My original circuit used a transistor connected to a data pin of an Arduino, with this data pin configured as the transmit pin of a serial port. As the serial data was transmitted from this pin, the transistor turned on and off an IR LED to match the serial data output. Below is a schematic that shows on the left the IR transmitter circuit and on the right the IR receiver circuit. In Transmitter Circuit – Arduino2, the transistor Q1 is connected to the transmit pin of a serial port and switches the IR LEDs on and off. I have added a second transistor, Q2, and this transistor is switched on and off at a constant frequency of 38 kHz. Because Q2 is in series with Q1, the serial data from Q1 is modulated by Q2 at 38 kHz. Now the serial data is transmitted on a 38 kHz carrier and can be detected by the IR receiver circuit, Receiver Circuit – Arduino 1, on the right-hand side of the schematic below.

The IR receiver circuit is taken from the TSOP382 data sheet. The components R1 and C1 are recommended to prevent electrical overstress.

I have several Raspberry Pi’s on my network that run headless, i.e. without monitor or keyboard. When I want to run an XWindows application on the Raspberry Pi and have the window appear on my Ubuntu desktop, I run ssh (secure shell client) with the -X option set, and the application’s connection to the X11 display is automatically forwarded from the Raspberry Pi to my desktop’s XWindows server. The connection is encrypted as all ssh connections are.

The command is:

ssh -X user@hostname

It’s that simple. The -X option takes care of setting the DISPLAY environment variable for you.

Note that man page for ssh mentions that enabling X11 forwarding should be enabled with caution for reasons of security. For this X11 forwarding to work, the remote machine has to make a connection back to the local X11 display, and an attacker could take advantage of this connection to perform activities such as keystroke logging. Make sure you trust the machines you use X11 forwarding with.

My first job in the late 80’s, early 90’s was working at a small start-up company, as part of a three-person development team. The product we were building was a Point-of-Sale (POS) scanning system, targeted at large, high-volume grocery store chains. I was responsible for the software that ran on the “front-end” register used by the cashier and developed this application in Turbo Pascal. The other two members of the team were responsible for the “back-end” store system, running UNIX and written in C.

At this time our company was breaking new ground; there were practically no PC-based grocery scanning systems in existence, and companies like IBM and National Cash Register (NCR) were just developing POS hardware that could run Microsoft’s DOS and software developed in high-level languages like Pascal and C. We practically had to design and build all of the software components ourselves – the only commercial library we used was for our database. We also had to develop our own polling protocol for the RS-485 network that connected the registers to the UNIX server. RS-485 is a serial hardware implementation similar to RS-232 but it allows “multi-drop”, i.e. multiple devices to be connected to the same wire at the same time. Each register was assigned a unique number and the backend server broadcast the registers’ numbers in turn over the network, and a register would respond with its request when its number was called.

The supermarket chain we were targeting as our first major customer had large stores that had upwards of twenty-five registers. It was critical that our network, back-end server, and front-end registers could cope with the transaction volume responsively and reliably. For a supermarket usually the only interaction a customer has with the staff is at the register, and so the check-out process has to be smooth and painless. We had signed an agreement to pilot our software and IBM’s hardware at a local store, and part of the plan was to send a busload of cashiers and a pallet of food to our office warehouse, to test the store system we had set up on folding tables. The system worked for the most part but we did have issues and a second round of manual testing was required. We did our first store installation and were successful; our small company won a $15M contract to install our POS software and IBM hardware in the rest of the supermarket chain.

Our existing testing process for a new release of software (and we did release often in the early days), was to have our customer support representatives and installers, at most five people, perform “ad-hoc” testing for a couple of hours. Our customer support team leader had worked in the supermarket industry before and would perform more systematic testing, but we were not testing the loading on the server and we had no repeatability in our test environment.

Several months later, after installing a handful of stores, we had reports of performance issues at one of the new sites. Our performance goal was to be able to poll each register, and if the cashier had scanned an item, return the item price and description, all within 0.3 seconds. My boss was considering setting up another store system in our warehouse, which required IBM shipping us loaner equipment, or packing up our development systems so we could camp out at the store and try to measure or recreate the performance issue.

At this time I carried a pager on week nights and weekends, and was getting “feedback” on the system issues our customers were experiencing. I had looked at the code and through inspection was able to find some possible defects, but what concerned me was that these defects were slipping through our testing. We had no way of testing the performance of the system on a large scale and our customer, Save-On-Foods, was also very concerned that our small company would not be able to fix the issues that were affecting its customers.

My solution for our testing gap was to create a “register simulator” that provided automation; I modified the register software to be able to respond to polls on the network for either itself or a group of registers and respond to the polls using input from a text file. I also added a “key-stroke logger” so that our support people could record their testing sessions as they created sales on the register, and the resulting text file could be copied, edited, and expanded. Now one register could impersonate a fleet of registers, and we had the beginnings of a repeatable test process.

With the new register simulator we were able to instrument our back-end server under load and solve the performance issue encountered by the newly-installed store. The register simulator also allowed us to tune our protocol for better responsiveness. We were polling each register in the store in a round-robin fashion, but depending on how busy the store was, many of the registers would be idle for long periods of time. One improvement was to create a “fast poll list” for active registers and a “slow poll list” for registers that were idle, and we would poll the fast poll list more frequently than the slow poll list. We were also able to simulate the effects of new message types on the network. For example, when I interfaced to the debit card networks, I found that to send all the data required by the bank in one packet required too much time and starved the other registers from receiving polls for their requests. I subsequently sent the debit card request in several packets.

While automated testing and simulation are not radical concepts, at the time there was no World Wide Web or Google to search for solutions. If the register was slow, it was my responsibility to lead the investigation. My innovation had a low cost of implementation suitable for a start-up company that needed to be agile, and helped to rapidly mature our ad-hoc testing process. With this innovation we were also able to solve some of the difficult performance issues that negatively affected a customer’s experience in the store, and regain our credibility with our own customer.

I wanted a simple mechanism that would allow me to send data to some of my Arduino (http://www.arduino.cc/) projects – I have a scrolling LED matrix display where I want to update the text message without having to hook up a keyboard and I wanted to be able to do this cheaply. Browsing through the web-site of a local electronics store gave me the idea of using infrared (IR) light – I could buy an IR LED and an NPN photo-transistor for a couple of dollars. I didn’t have a complete data sheet for the IR LED, only the information that it emitted light at 840 nm, outside of the range of visible light.

You can buy specialized IR receivers that are used in remote controls for televisions and other consumer electronics; these devices only pass through signals that are modulated on a 38.5 kHz carrier wave for purpose of minimizing interference from IR light from indoor lighting and the sun – it is extremely unlikely that a natural source of light would be modulated. Of course to use these specialized receivers, you need to transmit your signal on a 38.5 kHz carrier wave. Unfortunately the electronics store did not stock the specialized IR receivers and I would have to make do with the the simple IR photo-transistor they stocked.

My plan was to to connect two Arduinos together using a serial interface but instead of using copper wire to connect the transmit pin of one Arduino to the receive pin of the other, I would use IR light. I designed and prototyped the circuit below one January evening and I was excited to see that my receiving Arduino successfully echoed the “Hello world” message the other Arduino was transmitting.

I recorded the schematic of this circuit in my notebook and moved onto some projects. Eight months later I thought I should finish this project and write a communications protocol to ensure the successful transmission of uncorrupted data. The first step was to find the parts and breadboard the circuit, to start where I left off. Below is the schematic, prettied up by drawing it in Eagle CAD (http://www.cadsoftusa.com/) :

The photo below shows the breadboard with the two circuits. The Arduinos are out of the frame:

It didn’t work! I checked and rechecked my schematic against breadboard’s wiring and I couldn’t see anything wrong. At one point the the receiving Arduino did echo a “Hello world” message but only once. I was confident that the software was not the culprit – my background is software development and I knew the software logic was fairly simple. It had to be the hardware. But where to start? I know from my experience in debugging issues in software systems that you need to take a systematic approach. My only real hardware diagnostic tool was a $15 multimeter and my assumptions. I ran through the following investigation:

1. Both the IR LED and photo-transistor look identical; they are manufactured in a 5 mm clear dome packages, with a flat spot on the side of the casing to indicate the negative pin, i.e. the cathode in the case of the LED and the emitter in the case of the photo-transistor. Could I have mixed these components up? How could I tell? Both components were stored in their original labelled bags but I could have accidentally mixed them up when I tore the breadboard down six months ago. IR light is not in the range of the human eye and so I couldn’t be sure that my IR LED was really emitting light. I recalled that digital cameras can detect the wavelength of IR LEDs and when I looked at the view screen of my Sony Cybershot pointed at the breadboard, I saw the IR LED glowing with a bright purple colour. Okay, I had the correct components.(You can try this yourself – take your TV remote control and flash it towards your cell-phone or digital camera – you should see flashes of light in the view screen.)

2. Was my photo-transistor working properly? Could I have inadvertently burned it out? I connected the leads of the photo-transistor to the multimeter and selected the ohm-meter range. The photo-transistor showed a low resistance, even though I had not pointed the IR LED at it. This was my problem – I had somehow through my careless handling shorted out the photo-transistor. I covered the photo-transistor with my hand and suddenly the multimeter showed infinite resistance – the photo-transistor was working! Could I have transcribed the circuit incorrectly in my notebook?

Looking again at my notebook, I saw that the date I recorded the schematic was January 7th, 2012. I likely had created the circuit after work, after the sun had already set during the short winter days. I realized that I was trying to recreate this circuit during a bright summer morning, when I have a large IR light source shining through my window. I replaced the photo-transistor in the circuit and shielded the IR LED and photo-transistor with my cupped hand, and soon saw the “Hello world” message being successfully echoed on the serial monitor by the receiving Arduino. I made a shield out of heat-shrink tubing to block out the ambient light between the IR components and the circuit continued to communicate without my manual intervention. Below is the shielded version of the IR communications link:

For now I am going to use the heat-shrink tubing as my solution to reduce IR interference.

The code for the transmitter is shown below. I am using an ATTiny85 chip and the SoftwareSerial library with the Arduino 1.0.1 IDE: