Guitar Heronoid

Why play Guitar Hero when an android can do it for you??

What is Guitar Heronoid?

An android that plays the Guitar Hero game. A computer processes the video feed from the PlayStation2, detects where, when and how to play and moves the fingers accordingly.
This project was done by GarageGeeks members Rafael Mizrahi and Tal Chalozin.

Tuesday, March 20, 2007

The robot is divided into two basic parts, "brain" and "body".The "brain" is in charge of detecting and analyzing the PS2 video signals.The "brain" generates play-commands (which tone to press, when to do it and when to strum) for the "body".The "body" reads the play-commands and controls the fingers movement by them.

Some basic features were required:

No hacking on the guitar nor the PS2.

The hand should have a "look-n-feel" of a real human hand.(AKA, the fingers should fully move, just "like humans do")

It have to work fast enough, in order to be able to keep up with the song.

The "brain" and the "body" should be fully separated.

All the construction have to be detachable.

Parts (chronologically ordered):

Listener

As I said earlier, the "brain" generates the play-commands and sends them via TCP to the "body" listener. The listener translates the play-commands to a proper BYTE sent from the parallel to the body main control board.

Control Board

The control board is in charge of the getting the play-commands feed from the computer and delivering them to the fingers engines.

The feeds are getting directly from the parallel port, which generates not more then ~3.5volts. The hands solenoids needs 7VDC and around 800mA to operate. So, current amplifier was needed.

The control board is contains a seven-darlingron-array, where each input pin is connected to one of the data bits on the parallel post (pins 2-7). It's responsible for the current amplifying needed for the solenois which were connected to the darlington's output pins. Our mistake was the we haven't used the right darlington. The solenoids consumed 800mA but the darlington could manage only till 600mA. This caused to burn Refael's parallel port on his laptop….

The lack of time till the first robot live show, I built a new board using the same darlington but instead of connecting straight to the solenoids, I used relay to made the extra current amplifying.

Hands

The hand are the most important and complicated part of my job. I needed to build it to as close I can to a real hand (a robotic one). The main barrier was to be able create a real-like finger movement, and still pressing the guitar buttons.

It may look easy when a human holds the guitar, but the buttons are quite hard to press. You need to generate quite a lot of power in order to made them pressed.

The left hand is the "notes-hanler", I needed to make five finger hand.

The right hand is the "strummer", here I only needed one moving finger and four frozens.

The hands were made from aluminum "sculptured" by hand (w/o any use of heavy machinery). Each finger was connected to a solenoid that was hidden inside the palm.

(it's hard to see, but here are 5 solenoids,connected by wires to the fingers)

Both hands are completely detachable.

Doll

We wanted to create the feeling of a full-scale humanoid (human-android) is playing the guitar. First I thought basing it on a skeleton (like the one form the biology labs) will be cool, but I encountered too many mechanical difficulties.

The other option was using a display-window doll. A cool friend (that by incidence is the son of the owner of one of ISRAEL bigger clothing company (Honigman) ) was kind enough to give us a brand new display-window doll free of charge.

The problem with the doll was that his hands weren't in the right position (the elbow's angle wasn't changeable), so I needed to make them a changeable ones…

After cutting the right elbow, I inserted pipes for the control wires going from the control board to the hands and casted whipped polyurethane to make it permanent.

(from topleft->clockwise,cutting the hand; see the handicapped doll;pipes and wirings; getting it back together)

Each song is presented on a set of five columns, resembling a real guitar fretboard, that scroll constantly towards the player. The five columns correspond tothe five fret buttons and appropriately colored notes appear in these columns.

Notes detection area

Detecting the notes could be accomplished by using several approaches to detectmoving objects; most of them might not be fast enough to implement and use. Sowe came up with this idea: Looking at the game you can quickly realize that theimportant information, such as the plates is brighter than the rest of theimage.

HSV color space

Representing the color of the pixels in each trapezoid from RGB as HSV (Hue,Saturation, Value), also known as HSB (Hue, Saturation, Brightness), anddefining a threshold (Brightness Threshold at the properties dialog),somewhere in the middle of the Brightness value, gives you a binaryrepresentation of the pixels inside the trapezoid. Now, when the trapezoid isfilled with enough white pixels, there is probably a plate over there.

Binary black and white image

We connecting the game video output using a capture device into a computer. Alive video streaming filter captures the video frames as images and sends eachimage into the image processing part of the program that detects the notes.

one Robot Server and one Image Processing

Delays, Delays, , , DelaysTo play a note, the player must hold the correct fret button and press the strum bar. After playing and watching the game, you find out that the PlayStation add another delay to the equation. Pressing the frets buttons is recognized by the PlayStation game within some 100 milliseconds or so. Having this delay along with the delay it takes for the strum solenoids to go up and down, I realized that sending fret notes together with the strum action is not possible. I divided the protocol into two main actions:The first, at the area of detection, pressing one or more of the fret notes and lift up the strum. The fret notes information is added into a FIFO queue. The second, after about 250 milliseconds (Strum Delay at the properties dialog), pop the fret notes from the queue and send them to the guitar with a strum down.

A nice TODO is to detect the BPM of the song at the first seconds of the song, and adjust all those delays according to that.

Sending notes to Robot Server

Multiplayer is split-screen. In a "dueling guitars" fashion,two players tackle segments of the selected song. Unlike other modes, it is notpossible to fail a song in multiplayer, but scoring dictates that one playerwill generally win.Just by moving the area of detection to the side of the screen, using smaller trapezoids, all parameters in a different configurations file (guitarhero.ini), and there you go, you can play with or against the GuitarHeroNoid.