Super MIDIo Bros

Super MIDIo Bros.

Conception

Eashwar arrived to the hackathon after the other three members of the team had already settled into their spot (right next to the snack booth, conveniently). He had brought his MIDI keyboard with him since he came directly from a meeting for the Video Game Music Club at the University of Michigan (of which he is a leader). The brainstorming discussion we held started to revolve around the MIDI keyboard and what possible hacks there could be involving it. We settled on the idea of controlling Super Mario Bros. using a MIDI keyboard after Eashwar started playing the Mario theme on the keyboard.

The Beginning

The first thing we had to do was make sure we were able to read in the MIDI input from the keyboard. We experimented around with different libraries and even just directly reading from the USB buffer, but after some exploration we settled on using pygame.midi for our MIDI input. We also settled on pyautogui as the library to use to give Python the ability to manipulate the emulated NES game.

Biggest Roadblock

The most difficult challenge we faced was implementing multithreading into our hack. Since the pyautogui stuff would be holding down keyboard inputs for set amounts of time, we knew it would need to be its own subprocess. Initially, we tried to make the process an asynchronous call from our main MIDI parsing function, but that proved to be extremely confusing. Our next approach was to have the two processes as completely separate (launched separately by the command line), but communicating with each other through a pickled file. This also was a troublesome approach as file I/O was difficult to deal with. Finally, we ended up using the threading and queue libraries to allow for easy communication between the two processes. This proved to be succesful.

The Piano GUI

The final element was the keyboard GUI that indicated to the user what notes on the piano they needed to press in order to play the melody. We conveniently used pygame to draw this UI. We also took advantage of the fact that the piano is a 12 note repeating scale to do some modular arithmetic and calculate where to draw the indicating rectangles.

Built With

Try it out

Submitted to

Created by

I implemented the Piano GUI seen that instructs the user on what notes to play. I also worked to implement multithreading in the application to allow for us to send keystroke instructions to the NES emulator.

I worked with pygame.midi to retrieve MIDI events from the piano and return the notes played to Mackenzie's algorithm. I also wrote the PowerShell script (my first time working with PowerShell) to make our demo begin with one click, and to close the NES emulator when the Python process exited.

I wrote the majority of the driver code for the program, taking in the MIDI key presses and converting them to the correct keyboard inputs into the emulator depending on which set of notes the key presses matched.

I worked on how the computer simulated certain key presses given the input from the MIDI keyboard. This was my first time using Python, and figuring out how to simulate the keystrokes needed to make Mario move fluidly was a very fun challenge.