Revision as of 13:34, 1 November 2012

In this lab we will explore different possibilities for computer graphics in musical interaction using Satellite CCRMA, Pd, and a custom graphics system called SimpleGraphics. You need your MaxKit, with Satellite CCRMA.

Contents

Choose A Place To Sit in the Ballroom

For this lab we will be using the extra monitors in the ballroom for graphics display from the Beagleboard.

It is a little bit trickier to use the ViewSonic monitors because you will need to run the following extra commands on your kit before it works (so you might prefer to sit at a SAMSUNG monitor for example because then that won't be an issue)

So, if you are using a ViewSonic monitor, then please run the following commands on your kit

cd /boot/uboot

sudo pico boot.cmd

change line that starts 'setenv dvimode' to 'setenv dvimode 640x480@60'

Unzip the files directly on your board: change into your ~ directory and unzip the archive by running the following two commands:

cd ~

unzip graphicslab2012.zip

Remove the archive file by typing

rm graphicslab2012.zip

Change to the ~/graphicslab2012 directory by typing

cd ~/graphicslab2012

Install the graphics software by running a script (this will take a while and require entering the password temppwd):

./install-SGX.sh

If the installation was successful, then remove the graphics software archive:

rm GFX.zip

and then reboot your kit by running the following command (after this you will of course need to login again):

sudo reboot

SimpleGraphics

SimpleGraphics is a small program that can receive information about graphics primitives and then render these using OpenGLES. The result is a quick and easy visualization tool for environments that otherwise might have difficulty rendering real-time graphics, such as Pd on Satellite CCRMA.

To get started with SimpleGraphics, connect your Satellite CCRMA kit to a monitor and run the SimpleGraphics program on your kit:

cd SimpleGraphics/
./SimpleGraphics &

At this point you should just see a blank screen, because we have not given SimpleGraphics anything to draw yet.

Open up "simplegfx-primitives.pd" in pd:

pd simplegfx-primitives.pd &

Follow the instructions in the patch.

You can see that each drawing "command" consists of

send /sg/somecommandobject-namea-bunch-of-numbers

The command either creates or modifies an object that has that name, according to the parameters in the remainder of the command. For the ellipse command, these numbers correspond to

will create a rectangle a position (0, 0) with size (0.1, 0.1), with a color of (1, 0, 0, 1), i.e. pure red.

Play around with these primitives a bit. What command will make a circle that fills the whole screen? Try combining multiple primitives to create some more complicated image, like a keyboard or a face. What happens when you change a shape's alpha to less than 1? What if you draw multiple overlapping primitives with alpha < 1?

Animation

Close simplegfx-primitives.pd, restart SimpleGraphics, and open simplegfx-animation.pd. Here you will see a more complicated patch that sends out the same command periodically with parameters that vary according to the value of a sine wave. Note that osc~ is used to create the sine wave, just like for a regular audio signal, but we have to use snapshot~ to convert the value into something we can stick in a SimpleGraphics message.

Animation in computer graphics is useful for all sorts of reasons, but two stand out for use in interactive computer music. Firstly, obviously entities in the graphical space will change position and other parameters in response to human input, such as a circle that bounces around the screen when you shake an accelerometer, which we can accomplish using animation. More subtly, animation can be used to make static objects like "lively" and "natural". If in the previous example the circle's size modulates a little bit over time, it becomes more clear that this circle might be something we can interact with.

Come up with your own animation using SimpleGraphics and Pd. You might try varying the color of an object with noise~ -- do you want to vary the color across the full spectrum, or just add a little "crackle" to a static color?

Combining Sound and Graphics

Again restart SimpleGraphics, and now run simplegfx-sound.pd. You'll see this plays back a short audio file, mapping volume to the size of a graphical object and pitch to its color.

The issue of mapping is crucial when combining audio and visual media, as individually music and visual arts don't always have a natural correspondence.

Try changing this patch to map volume and pitch to different parameters of a different type of object. Does this make more or less sense to you than the original mapping?

Make A Musical Interaction

As the final main deliverable of the lab, create an interaction with the Satellite CCRMA kit that produces both sound and graphics in response to sensory input. Think about what you want to design before you get started, and especially consider the relationship between the generated sound and graphics.