112 - Leap motion auditory overload (LMAO)

Inspiration

We were thinking how fun it would be to DJ in the air

What it does

The system has 2 modes the first is a game mode, where players get move their hands in a figure 8 configuration to match the beats of the music, basically a rhythm game. The second part is the freestyle mode, which controls the tempo and speed of the music using the velocity and position of your hands which is determine using leap motion javascript API.

How we built it

We build it with rainbows, candy and alot of code :P. No lah, essentially, it is build on top of leap motion javascript API , HTML5, alot of CSS and some python scripting (to mine the BPM of mp3 files and audio processing). The javascript api basically convert infrared computer images that the leap motion sensor detects, into useful data points like position of left or right hands. From these data points we control the music or pointing system of our application.

Challenges we ran into

There were alot of challenges we ran into, one of the challenges was trying to determine whether your hands pass through a yellow dot to count as one point of our system, which we solved using reference point system and some basic math. Another issue was trying to slow down music without affecting pitch which JS doesn't easily do, and in fact is a non-trivial problem in audio processing. We did this be pre-rendering the files using aduacity and python scripting. Another challenge was correlating the velocity of the hand to the tempo of the music, this was particular challenging due to the limitation of the sensor and the new concepts in computer vision we had to learn. Finally, and most importantly, learn JS we picked up JS only a few days before the hacknroll and some of us picked up even at the hackathon itself.

Of course there are many more like the switching between the audio files and even the animation of the entire game, but you will see them when you come to our booth.

Accomplishments that we're proud of

Overcoming the challenges we faced and freaking learning JS.

What we learned

JS , HTML, CSS and alot of web tech.

What's next for Leap motion auditory overload (LMAO)

Probably create a feature to control the pitch of the music and maybe even add a composer mode where uses can control the weights of a generative neural net to make music instead of only controlling it.

Created by

I worked on the JavaScript parsing the velocity data of the palm to determine the suitable beats per minute at which the song should play - allowing the song to play faster or slower based on the movement of the hand.
I also assisted in the rendering of the audio files.