Shakey robot plays Angry Birds

At this year’s Pycon [Jason Huggins] gave a talk about his Angry Birds playing robot. He built a delta robot which includes a pen actuator for controlling a capacitive touch screen. The video after the break starts with a demo of the bot beating a level of Angry Birds on the iPad.

The idea behind the build is that robots like this could be used for app testing. I this case [Jason] has tweaked the servo commands manually to achieve the results. But during the talk he does demonstrate some machine vision to analyze and win a game of tic-tac-toe.

We do enjoy seeing the robot, but we’re not sold on the thought that testing will use robots. Perhaps there is a niche need for this type of thing, but we assume the majority of automated testing can be done in the emulator for the device on which you are developing. What we really want to know is how the capacitive stylus works. We didn’t catch him talking about it at all. We want a reliable, yet simple way to electronically trigger touchscreen inputs (along the lines of this project). If you know how [Jason’s] stylus is working please share your thoughts in the comments section.

The stylus uses a piece of conductive foam, like the stuff ICs are packed in. A small piece will not trigger the screen, but when switched to ground (with something like a FET) or connected to a piece of metal (even a longish wire) it has enough capacitance to simulate a touch. This stylus has a piece grounded by a wire on the end of the pen.

It uses a piece of conductive foam (similar to what ICs are packed in). When switched to ground or connected to a large enough wire, it has enough capacitance with the screen to trigger a touch. If it is just a small piece of the foam, it doesn’t trigger. This stylus has a piece of the foam on the tip of the pen, but it it connected to ground via a wire.

if you have enough metal you don’t need anything else. A conductive foam tip provides a non-scratching contact point with the screen. I have tested this by insulating myself from different materials and using them on capacitive-touch devices. Batteries tend to work particularly well.

(@ Mike Szczys) [quote]What we really want to know is how the capacitive stylus works. We didn’t catch him talking about it at all. We want a reliable, yet simple way to electronically trigger touchscreen inputs (along the lines of this project). If you know how [Jason’s] stylus is working please share your thoughts in the comments section.[/quote]

Maybe it’s just my eyes, but it would appear that the “hand” of the delta bot has a capacative touch stylus attached to it. I believe they use a conductive rubber tip for those, maybe carbon based? I might be completely wrong though.

However, in my early prototypes, I simply used anti-static foam that chips are often shipped with. I then stuffed the foam into a hollow brass rod. While the foam/rod solution was way cheaper, I found the Targus stylus was more reliable — the foam kept falling apart or falling out of the tube. In the future, I hope to get my hands on some some conductive rubber and roll my own more durable stylus.

In either case (foam or commercial stylus), I also had to attach a grounding wire. I don’t understand all the physics involved, but I needed that grounding wire to get the touch to properly register on the screen surface.

Also regarding:

but we assume the majority of automated testing can be done in the emulator for the device

Yes, most testing should be done on an emulator, but in the case of Android for example, there is a huge difference between emulator and real device performance. Developers would rather see how their app behaves on a real device. Combined with the fact that new devices are coming out all the time and that proper testing APIs often lag the initial release, a ‘robot finger’ is useful when the only other option is manual testing.

Robots are routinely used for testing the UI of cellphones. I worked on a custom system which pressed buttons and had cameras for spotting dead pixels and LCD artefacts using a neural network. It was used on a production line back in the mid 90’s the brains were a rack full of DOS PCs.