I found myself needing to verify caller ID operation of a couple of troublesome telephone handsets. To properly test, I want to run 1,000 calls to the handsets and verify they decode caller ID every single time.

During early testing, I would manually call the handsets about 20 times to see if they looked good. Typically the error occurs every 3rd-4th call so that was a little slow but do-able. But there is no way I’m sitting still to watch 1,000 calls.

I could video record the handsets, but watching the video is no faster and trying to fast forward was going to be clumsy as well.

What I really need is a way to toss out most of the video frames then I could just advance a few frames to see the next call.

My first test was to extract one frame per second which I did with this command:

ffmpeg -i myfile.mp4 -vf fps=1 c:\tmp\pics\out_%%03d.png

This created a PNG file once a second (1 frame per second), reading myfile.mp4 and creating a series of PNG files called out_001.png, … .

This worked pretty well, but a complete test cycle takes about 20 seconds so I was ending up with 20 PNG files / cycle to flip thru. That’s too many.

I watched the testing process, and both phones’ displays are on for about 10 seconds from the time the CID message comes thru. If I were to capture a frame every 5 seconds, that would be about perfect.

So I ended up using this command:

ffmpeg -i myfile.mp4 -vf fps=0.2 c:\tmp\pics\out_%%03d.png

Now I end up with about 5 frames per test call.

Using picasa to view the first file, I can go to the first picture showing the caller ID message. Then I just press the right key 5 times and I’m on the same spot in the next test. That I can do very quickly.

I can process roughly 1 test every 2-3 seconds flipping thru 5 pics verses having to watch hours of video which just isn’t going to happen.

Update

I’ve went thru this process quite a few times now. Works well, but having to constantly click thru still images is a pain.

I created a video to watch. I can then stare blankly at the video and just pause when I think an event has been missed.

When I did the extract, I was extracting 1 frame from the original video every 10 seconds. When I create the new video, I want to see each of those extracted frames every 1/2 second (2 frames per sec). This is the command I used to construct the video:

I needed a new RPI for a project and found version 2s are more expensive than version 3s so I purchased a 3 to be my new dev machine and I’ll use the old hardware for the new project.

The problem with upgrading my dev hardware is the last thing I want to do is screw with the software on it. It has taken a lot of time to get it where I want it. So it is important that I can just transfer the SD card and USB drive to the new hardware.

Here is the procedure I followed which worked just fine. I won’t waste time describing each step blow-by-blow. If you need to know the details, you can google them as I did.

Backup The System

If something goes wrong, you will want a backup. Expect to spend a few hours backing everything up (at least if you have a very big USB drive).

My RPI Version 2 was running Raspian version Wheezy. You MUST be running Jessie (or later) OR do the following dist-upgrade.

As I understand it, dist-upgrade doesn’t upgrade you to Jessie, but makes sure you get ALL available packages (apt-get upgrade may hold back packages). My RPI 2 is using Wheezy, I followed this update and it worked fine for me.

Update the list of packages:

sudo apt-get update

Do the dist-update:

sudo apt-get dist-upgrade

Reboot to load the changes:

sudo reboot

Make sure things are working, then power off your RPI 2:

sudo poweroff

Upgrade The Hardware

At this time, pull the old RPI2 out. Transfer it’s SD card and connections to the RPI3. Power the RPI3 up.

Been working on a complicated teensy project that is going to the other side of the state for testing. So I’ve been building a testing platform that consists of a Raspberry Pi connected to the Teensy via USB. I can connect to the RPI and monitor/update the Teensy.

One of the requirements is that I be able to cycle power on the teensy just in case duty hits the fan. I’ve got this great little device for that very purpose:

The one problem I’ve realized with this idea is I cannot just remove power from my device to reset it because the RPI will continue to feed power to the Teensy via the USB cable. (Actually there is a way to have RPI kill power to its USB hub but it also kills power to the ethernet module. It has worked in testing but seems a little dicey to really rely upon).

I need to kill power from the USB cable. The obvious way to do this is to slice the USB cable open and cut the power line. Not too hard to do, but I really don’t want to send a unit out to the ‘customer’ with electrical tape around one of the cables. That’s not cool.

I was looking around for data only USB cables (which are available but kind of expensive) and stumbled across someone that was doing the reverse of what I wanted – he was getting rid of the data signal. BUT, his method was perfect. See http://www.instructables.com/id/USB-Condom/?ALLSTEPS.

I applied the same idea and cut a tiny piece of clear packing tape and applied it to the +5V contact on the A side of the USB cable:

You should be able to see the tape covering the contact on the far right.

This works great and now when I kill power to the device, it actually turns off.

This post covers how to remotely monitor / upgrade an Arduino or other MCU behind a firewall.

Let’s look at an example:

On the far right of the diagram is an Arduino ‘in the field’ and I want to access it from a local PC which is on the far left.

The first step to accessing the remote Arduino is to connect it serially to a networked computer. I propose using a Raspberry Pi for this purpose since they are cheap. I then wish to use putty and VNC to access remoteRPI to finally access the Arduino. But just adding remoteRPI doesn’t make either accessible.

There are a couple of problems with this solution. First, remoteRPI will most probably be assigned a private IP address by DHCP. There is no direct access of remoteRPI due to the remote firewall, remoteFW. A hole is going to be needed in the firewall.

Putting holes in ‘customer’ firewalls is a difficult proposition, at best. Having the ‘customer’ create the hole is probably impossible. Gaining access and doing it yourself is equally impossible.

The usual way to get around the firewall access issue is to have the remoteRPI create the connection back to the local network. The remoteFW has no issues with outbound connections. We create and use that connection when necessary.

Using remoteRPI to create the connection is the access method I will implement here. This will be done using an SSH reverse tunnel.

At boot time, remoteRPI will open an SSH reverse tunnel to localRPI. I’m using a Raspberry Pi for localRPI, but any linux system will work. This tunnel remains open continuously. When you want to gain access to remotePRI, you make an SSH connection to localPRI which forwards to remotePRI.

I was unable to find any references that fully showed implementing the diagrammed solution from begin to end. This blog will cover all of the config changes and commands to implement this tunnel but not the whys. You can google all of the commands and learn the why easily enough so I won’t try to duplicate the effort.

Allocating TCP Ports

We are going to need to define 3 TCP ports for our project. I like to use a random number generator (google random number generator) to select a port between 10,000 and 32,767.

27045 – Local Firewall NAT Port

This is the port that will allow remoteRPI access to localRPI via SSH (port 22).

27046 – SSH forwarding port

We will connect to this port on localRPI when we want to access SSH on remoteRPI.

27047 – VNC forwarding port

We will connect to this port on localRPI when we want to access VNC on remoteRPI.

Create NAT Rule on localFW

remoteRPI needs to connect to localRPI via SSH (port 22). We will create a port forward rule on the localFW to allow this access; however, we will expect port 27045 to be the destination port on the firewall and we will translate that to port 22:

Application

Protocol

Source Net

Port From

IP Address

Port To

sshRevTun

TCP

27045

192.168.0.33 [localRPI]

22

This rule states any inbound packet with destination port of 27045 will be sent to IP address 192.168.0.33 (localRPI) using destination port 22.

Activate the firewall rule.

You can then do a simple test on the rule by using a remote port check utility such as http://ping.eu/port-chk/ and enter the external IP address of your firewall and port 27045 to verify the port is open.

Enable Global Port Forwarding

By default, any reverse SSH tunnel built on localRPI will only be available to localRPI. My requirement is that I be able to use any PC on my local network to access remoteRPI, so we need to allow others the ability to use the reverse SSH tunnel.

On localRPI, in the file /etc/ssh/sshd_config, add the following line anywhere:

GatewayPorts yes

Save the file and restart SSH with

service ssh restart

Make Scripts to Create the Tunnels

We need two scripts to build the two SSH reverse tunnels – one for each protocol (SSH and VNC).

I create these using the standard pi user and simply place them in pi’s home directory.

These scripts will connect to localRPI by routing to the localFW (localfw.mydomain.com) and then being forwarded by the firewall to the localRPI system where they will attempt to log on as the pi user.

I am using a DNS name localfw.domain.com. You can use a DNS name or the external IP address of your router as well. If your router is assigned its WAN IP address using DHCP, you should seriously consider using a dynamic DNS service such as afraid.org. Otherwise, when your ISP finally forces the IP address on your router to change, the SSH reverse tunnel will fail, and it will be be difficult to fix.

Test Scripts

You might want to comment out the initial sleep 60 command as this is really only necessary when the script is executed at boot time.

To start the SSH reverse tunnel for SSH, type

./sshRevTunSsh

You will be asked for the pi user password for localRPI. Then there will be no further output, it just runs.

Now from a local PC, run putty. For the hostname/IP address, enter the hostname for localRPI and specify 27046 as the port:

localRPI 27046

This will connect you to remoteRPI. Enter a user/password for the remoteRPI system.

Should the script fail, you can place -v in the SSH cmd of the script to get debugging output.

Now run the script for sshRevTunVnc. From a local PC, run VNC and login using

localRPI:27047

This will connect you to VNC on remoteRPI.

Create SSH Reverse Tunnel without Requring a Password

The scripts to start the SSH reverse tunnels allow automation, but having to enter a password prevents us from actually automating the process.

There are many locations on the web explaining this process. I will do a very brief example of it.

On remoteRPI, logon to the pi user and type:

cd ~/.ssh
ssh-keygen -t rsa

This creates a public/private key pair for password-free logins.

Now copy the public key to remoteRPI:

scp -P 27045 id_rsa.pub pi@localfw.mydomain.com:.ssh/authorized_keys

Rerun the scripts and they will no longer require passwords.

Fully Automate the SSH Reverse Tunnels

Everything is now in place to allow the SSH reverse tunnels to be created at boot time. Should the tunnel go down, the script will automatically restart it.

Finally, again test that you can access remoteRPI from your local PCs.

*****

I’m sticking this note here because I don’t have a better place for it yet.

Now that I’ve got the remoteRPI operational, I’m starting to experiment with actually controlling / updating an Arduino.

I started with a Nano and have found there is a problem with Nanos. When you first connect the Nano to the RPI via USB, it will come to life and populate /dev/tty properly and you can access it via /dev/ttyUSB<n>.

But, when the RPI is rebooted, the Nano disappears. Nothing seems to bring it back short of unplugging the USB cable and reconnecting. Not acceptable.

A weir gate is attached to swimming pool skimmers to keep the rubbish collected by the skimmer from floating back out into the pool.

The ones I’ve seen use a Styrofoam float to keep the gate up. I noticed my pool has been hard to keep clean lately, and I realized the old weir gate was no longer floating properly.

OK, I should say there is very little good reason to refurbish a weir gate because they are really cheap. Unless you are really cheap (nothing wrong with that), OR, as in my case, the new one you ordered doesn’t quite fit.

The weir gate I ordered was just a smidge too large. I was going to need to trim maybe a 1/4″ off. But first I needed to get the old one out which was kind of a mystery. All instructions I have found talk about installing the new one not removing the old one.

As I looked carefully at it I looked like I could get a very thin screw driver under the ‘post’ I was seeing and maybe pry it out. Sure enough I was able to work the ‘post’ out of its socket and I realized it wasn’t little pins holding the weir gate in place, but the posts set in shallow sockets.

I don’t know if they all work the same, but the weir gates I have have a rod inside them that is like a compression curtain rod as you can see from the picture.

If you can remove the weir gate intact, it is perfectly easy to refurbish.

Once the weir gate was out I could remove the Styrofoam and sure enough it was badly water logged. It weighed as much as a small brick (pictured at the top).

As I looked at the Styrofoam, I realized I had extra Styrofoam in my parts bins I’d saved from packing material – it comes in handy once in a while. I located my stock and it has the same texture as that in the new weir gate, so I decided to use it (leaving the entire new weir gate for future use should I need it).

I cut my stock down to the same dimensions (2nd block of Styrofoam above) and inserted into the weir gate.

Now to reassemble. The new weir gate has pins holding the compression rod in place. I had tiny picture nails of about the same size. I stuck one nail in on one side, then used a dowel to compress the rod into place so I could get the 2nd nail positioned (having a helper here is very, well, helpful).

Once the pins were in place my old weir gate looks just like the new:

Note the nails are sticking out on the flat side – this is the side you will be able to access with pliers once the gate is positioned properly.

At this point, installation is simple – position the gate so the posts are aligned roughly with the sockets. Pull the first nail, check alignment again, then pull the 2nd nail and you are done.

I’ve got a few Teensy projects in ‘production’ now and I know I’m going to have to update the software onsite. In fact, I need to be prepared to allow an end-user to upgrade the software as one Teensy that is located 1000’s of miles away now.

In the back of my head I figured this must not be too hard since the Teensy has a nice windows based programmer. Sure enough, it is slam dunk. Well, pretty close.

This procedure assumes that the necessary files will be gathered together, sent to a remote site, and a reasonably technical user there will be responsible for doing the install (not necessarily someone familiar with programming arduinos, but one able to do normal PC maintenance).

Put Together the Files

To update a Teensy in the field, you are going to need the Windows serial installer, the Teensy programmer application, and the binary code for your own program which consists of a .HEX file and a .ELF file.

Assuming the PC used to to the upgrade doesn’t already have the Arduino/Teensy software installed, we are going to install just the files necessary to perform the update.

Right click on the serial_install.exe file and run as Admin to let it install the virtual serial driver.

Connect to the Teensy

Plug the Teensy into the PC’s USB port.

Windows should see the device and start looking for drivers. Skip searching the internet for the update.

Once the driver is installed, there is no indication I can find that device manager sees the Teensy (it won’t update the serial port list). But you should still hear the sound effect when you unplug/plug the teensy in.

Update the Code

Start the Teensy.exe program

Click on Help | Verbose – You will see a screen which will indicate the program is ‘listening for remote control’.

In Teensy.exe, do a file | open and open the HEX file. The log indicates the file has been opened.

Press the ‘Auto’ button in teensy.exe.

Press the reset button on the Teensy.

You should see messages in the log indicating the download and reboot were successful:

Once I had this process working, it became evident I no longer have to recompile programs every time I want to use them.

This is most helpful on my current project. There are quite a few hardware subsystems involved. After I build a new PCB, I want to make sure all of the hardware is fully functional, so I have a hardware test suite I use.

I can now download and execute the hardware test suite w/o compiling. Once the hardware is tested, I can then download and execute the stable version of the software without having to go back to the source and recompile it.

Nice!

Nov 2016 Update:

Periodically, I have an issue getting this process to work. When I do a file | open in teensy.exe and specify the HEX file, I will get an error indicating file too large.

It seems teensy.exe defaults to an older model of teensy with less program space. Supposedly if you have both the HEX and ELF files available this isn’t supposed to be an issue, but sometimes it is.

It seems that if I build the HEX file by doing a sketch | upload this problem will not occur. But sometimes I don’t want to upload the version I’m compiling to the teensy that I have connected so I do a sketch | compile. I don’t know why that would make a difference, but so far my observations have been it does.

What to do when this happens? Today I seem to have stumbled across a solution. I have not been able to replicate the failure so I’m not 100% I got the correction sequence right:

After doing the file | load, you see the file too large error in Teensy.exe.

Double click on teensy_reboot.exe – this will upload the file anyway.

Then double click on teensy_restart.exe – this will restart the teensy.

I have to go thru my notes every so often to find the formula for navigating backwards, so I’ll post this stuff here in case anyone else finds it useful.

Modulus math is very handy for dealing with circular functions like time. In fact, I use this to blow the minds of kids (I’m always trying to pique their interest in math and other technical stuff).

I’ll say something like “When is 11 + 2 = 1”? They will insist it cannot be and I tell them they see that very computation on a regular basis. After I have them telling me there is ABSOLUTELY NO WAY 11+2 EQUALS 1, I ask them what they get when they add 2 hours to 11:00. 1 O’CLOCK!!

Anyway, when writing software it is really useful to use modulus math to solve these little problems. To get 1:00 in software, I’d write

x = (11 + 2) % 12
x will contain 1

If you aren’t familiar with the C % operator (a.k.a the mod operator in Pascal), in essence it returns the remainder. A / B (assuming A and B are ints) returns the quotient. A % B returns the remainder.

int(11 / 2) is 5
and
11 % 2 is 1

In fact, long ago when I wrote COBOL code, if I needed to do a MOD operation it was actually part of the DIVIDE statement:

When the program starts, it prints the first string in the list. It then waits for either an up or down button to be pressed:

// C like psuedo code
while (!done) {
lcd_out(list[i]);
if (!keyAvail()) {}
key = getKey();
if (key == keyUp)
// compute new i where we go forward in the list
i = i + 1;
else if (key == keyDown)
// compute new i where we go backward in the list
i = i - 1;
}

This works OK until we get to the end of the list. We need to wrap around when the user presses the up or down key.

Formula 1: Going forward from the last element to the first
i = (i + 1) % listLen

If i is 4, pointing to the last element in the array, formula 1 will add 1, giving 5. Then 5 % 5 is computed which is 0. We are now pointing back to the beginning of the list.

To go the other way:

Forumla 2: going backwards from the first element to the last
i = (i + (listLen-1)) % listLen

If i is currently 0, we add 4 (listLen-1), and get 4. Then 4 % 5 is 4, so we are correctly point to the last element in the list.

Formula 1 is easy to remember. Formula 2 I can never seem to make stick and have to look it up every time.

Here is the entire program in C like psuedo-code:

while (!done) {
lcd_out(list[i]);
if (!keyAvail()) {}
key = getKey();
if (key == keyUp)
// compute new i where we go forward in the list
i = (i + 1) % listLen;
else if (key == keyDown)
// compute new i where we go backward in the list
i = (i + (listLen - 1)) % listLen;
}

and Pascal-like psuedo-code:

while (not done) do
lcd_out(list[i]);
if (not keyAvail) {}
key = getKey;
if key = keyUp then
// compute new i where we go forward in the list
i := (i + 1) mod listLen
else if key = keyDown then
// compute new i where we go backward in the list
i := (i + (listLen - 1)) mod listLen;

The Arduino’s millis() function returns the number of milliseconds since the arduino started running. It returns an unsigned long, 32 bit, value.

A 32 bit unsigned word can contain 0 – 4,294,967,285 (2^32-1). When millis() gets to 4,294,967,285 milliseconds and adds 1, it ‘rolls over’ to zero. It will take 49.7 days (4294967286 / 1000 / 60 / 60 / 24) for this to occur.

If you want your Arduino project to run continuously for more than 49 days you will want to take this roll over into account.

Unsigned Subtraction

Getting around the roll over issue is handled by using the subtraction property of unsigned words.

I’m going most of my examples using 8 bit unsigned words (or bytes). These are declared as uint8_t and can have the values 0 – 255 (2^8 – 1).

You cannot have a negative number stored in an unsigned word. If you have an uint8_t and subtract a large enough number such that the result is negative what happens?

uint8_t x;
x = 0 - 1;

In binary, we start with zero

00000000

and subtracting 1 gives

11111111

or 255 in decimal. S0 0 – 1 is 255 when using uint8_t.

Here is the general algorithm the CPU is going to use to subtract two unsigned words, x – y:

if (x - y) < 0 then
z = x - y + 2^ws
else
z = x - y;

If (x-y) is positive, nothing special happens. If it is negative, we must add 2^ws where ws is the word size (we are currently using 8 bits).

Using this general rule, if you want to subtract 0 – 1, the result is going to be negative, so calculate z as

z = 0 - 1 + 256
z = 255

That’s exactly what was expected.

w = 255;

To be clear, YOU do not have do the above algorithm. It is done for you when you subtract two unsigned words:

uint8_t a,c,x;
a = 0;
b = 1;
x = a - b;
and x now contains 255 because were are using uint8_t

Examples of Roll Overs

Let’s watch how subtraction works as roll overs occur. Assume x is increasing and rolls over. Let’s subtract 10 from each value of x and see what happens before and after the roll over (again, this is using an 8 bit unsigned word):

as x approaches roll over (254 & 255) x-10 returns a positive value and nothing more needs to be done; however, once x rolls over, a negative value is calculated and so we must add 2^8 which gives us the correct answer thru the entire sequence: 244, 245, 246, 247.

Now imagine x is the current value of millis() and the constant 10 was the initial value of millis(). millis() – oldMillis returns the next proper value when millis() rolls over.

You may be looking at the x-10 result which is also nearing 2^32-1, and thinking that’s about to roll over anyway – this didn’t buy anything. This is only because I selected such a low value of 10 for the subtraction to keep the example simple. In real life this most likely will not be an issue (see The Catch).

Using Unsigned Subtraction to Our Advantage

Now that you’ve seen how unsigned subtraction works during roll over, let’s take advantage of it in a program.

We need to track the initial value of millis() when we start the timer and we need to know how long the timer should last. I typically use the names xxxTimer and xxxTimeout for these variables and define them something like this:

const unsigned long keyTimeout = 1000UL;
unsigned long keyTimer;

You start the timer by assigning the current value of millis() to it:

keyTimer = millis();

This is the important part: you must subtract the initial timer from the current millis() value and compare that to your timeout. As long as you do that, your program will handle a roll over just fine:

Well, this isn’t really a catch as much as a usage consideration. The word size of the return value of millis() limits the length of the timeout. You cannot have a timeout larger than 2^32-1 milliseconds (49.7 days) even if roll over does work properly.

In the 32 bit example above, I set the initial value of millis() of 10. When millis() rolls over, it will be just 10 msecs short of exceeding the maximum allowed timeout. So while the roll over works fine, the I’m about to exceed the maximum possible timeout.

Note on UL Constants

You may have noticed that I used UL on my constants to force their typing such as 1000UL. You may wonder why.

For simple assignments, this really buys nothing:

xxxTimeout = 1000UL;

However, I tend to use constant expressions so I can read the code a little easier. For example, if I want a timeout of an hour I will code it as:

xxxTimeout = 1UL * 60UL * 60UL * 1000UL;

(1 hour * 60 minutes * 60 seconds * 1000 msecs)

rather than

xxxTimeout = 3600000UL;

If I need to make a change I’ll see that expression and realize it is for 1 hour and if I need 2, I just change the first constant (since the expression is evaluated at compile time I loose nothing do this).

The problem I’ve had is the C++ compiler periodically will NOT compute the constant expression properly if I don’t typecast each constant. This has bit me enough times that I simply never leave UL off any constant (or really any other non-integer constant for that matter).

I cannot say what the exact circumstances are to cause this issue because in every case, by the time I figured out what was going wrong I was too irritated to research the root of the problem.