Speaking of the music video, let’s show it onscreen. While I first did this with a simple HTML <video> tag & positioned it on top of the canvas with CSS, I soon realized I’d want to work with it in the p5 context. Luckily, p5 has a createVideo function, although oddly it works nothing like sounds or images.

let vid, song, analyzer, fft

functionpreload(){

song =loadSound('makemefeel.m4a')

}

functionsetup(){

createCanvas(1024,768)

vid =createVideo('makemefeel.mp4')

vid.hide()

vid.pause()

vid.position(width /2, height /2)

vid.size(512,384)

ellipseMode(CORNER)

analyzer =newp5.Amplitude()

analyzer.setInput(song)

fft =newp5.FFT()

fft.setInput(song)

}

functionmouseClicked(){

if(song.isPlaying()){

song.pause()

vid.pause()

}else{

song.play()

vid.play()

vid.volume(0)

}

vid.volume(0)

}

functiondraw(){

background(0)

noStroke()

if(song.isPlaying()){

// Volume indicator

let rms = analyzer.getLevel()

fill(255)

ellipse(width /2, height /2,16+ rms *512,16+ rms *512)

// Frequency graph

let spectrum = fft.analyze()

fill('#fae')

beginShape()

vertex(0, height)

for(i =0; i < spectrum.length; i++){

vertex(i +1,map(spectrum[i],0,255, height,0))

}

endShape()

}else{

fill(255)

textSize(72)

textAlign(CENTER)

textFont('Futura-Bold')

text('CLICK TO PLAY', width /2, height /2)

}

}

Great! I discovered through this that the p5 video APIs are pretty basic & not well-documented, but we’re going to continue nonetheless…

The video & the rest of the sketch are really disconnected, though. What if we pulled colors out of the video to use in the background?

This seemed pretty tricky, but p5 has functions to help! This tutorial was super useful.

Here’s how they describe the getPixels() helper:

Pixel data in .pixels is arranged such that the red, green, blue and alpha values of each pixel are stored as separate items. The first four items are the RGBA values for the pixel at 0,0; the second set of four items are the RGBA values for the pixel 1,0; the next four are for the pixel at 2,0; etc. When a row of pixels on the screen ends, the pixel data starts over again at 0,1 (and then 1,1 and 2,1, etc).
The two for loops iterate from zero to the width and height in both dimensions. The expression ((y * width) + x) * 4 gives the offset of the four values that correspond to the color of the pixel at x and y. Then, I set the fill color to the red value for that pixel by getting the value at vid.pixels[offset], the green value for that pixel by evaluating vid.pixels[offset+1], and the blue value by evaluating vid.pixels[offset + 2].

Ok, that was intense but makes sense. I decided colored confetti will be this simplest implementation, since the particles can directly map to pixels, no filtering or processing needed.

I looked at their code for making a video pixelated & used it to write a confetti implementation. The main difference is that instead of lining up the pixels over the original video, we want them elsewhere on the screen, a basic version of which we can create with random(0, {width,height}).

Let’s first fix the black pixels issue. Basically, if the pixel’s RGB values are [0, 0, 0], we should cut it. So in the nested loops, we can skip any pixels where this is the case:

// Filter out black pixels

if(vid.pixels[offset]==0){

continue

}

Again, good start, but this has issues too. Like the bouncing ball in class, it’s leaving near-black colors, which aren’t great either. Since we need the pixel color for both the filtering & the filling, let’s pull it into a separate variable to simplify the code, too.

// Add confetti of current colors

vid.loadPixels()

const px = vid.pixels

for(let y =0; y < height; y +=16){

for(let x =0; x < width; x +=16){

let offset =(y * width + x)*8

// Filter out black pixels

const rgb =[px[offset], px[offset +1], px[offset +2]]

if(rgb.map(value=> value <24).includes(true)){

continue

}

fill(rgb[0], rgb[1], rgb[2])

rect(random(0, width),random(0, height),16,16)

}

}

Except…we also need to filter out white pixels. The first way that came to mind was to check if all the RGB values summed was equal to the sum of [255, 255, 255].

Now, JavaScript doesn’t include a sum() function in its standard library, and I want to avoid adding a utility library like Lodash for the time being. We can make use of the reduce operator, starting at 0, to calculate the sum, like this:

This is a lot simpler than the filtering. I switched rect to ellipse & added ellipseMode(CORNER) to the setup function to keep the coordinate system consistent with the rectangles we were using before.

Aren’t we getting a little side-tracked? Yes. But that’s how I work, so we’re not waiting to go down this road.

My first thought was to make the the volume indicator super small & in the corner. But that’s boring, & Janelle deserves better.

I had an idea in the meantime to saturate the colors more depending on the volume, but since I think that would require converting RGB to HSL, increasing saturation, & converting back again, let’s skip that for today.

More simply, what if we applied the concept of the volume indicator to the confetti themselves? It’ll be similar code for calculating the size, but have a smaller minimum size, and not give too much influence to the volume. Here was the code for the former volume indicator:

ellipse(width /2, height /2,16+ rms *512,16+ rms *512)

Attempt #1:

const size =16+ rms

ellipse(random(0, width),random(0, height), size, size)

Actually, rms is usually a value around .03xx, so let’s convert it to a small integer:

Dynamically finding a dominant color out of all the pixels would be memory-intensive & frustrating to code, but luckily my simpler idea first.

I set let dominant = [] before the confetti, then after the filtering, set dominant = [rgb[0], rgb[1], rgb[2]], and changed the frequency graph to use that color. It’s not an ideal implementation, but it’s 90%.

I used the dominant color to change the background of the paused screen:

if(dominant[0]){

background(dominant[0], dominant[1], dominant[2])

}

I centered the canvas on the webpage (using flexbox):

body{

margin:0;

padding:0;

display: flex;

min-height:100vh;

background:#000;

}

canvas{

display: block;

margin: auto;

}

But then made the canvas fullscreen:

functionsetup(){

createCanvas(windowWidth, windowHeight)

}

I added a rounded rectangular frame to the video:

// Add rounded frame

fill(0)

rect(width /4-48,128+40,768+96,512-80,24)

I fixed a few bugs, like the audio & video tracks playing simultaneously (set the video’s volume to 0), adjusted the confetti size some more, & stopped the filtering from generating particles with undefined fills.

This app doesn’t work well on mobile browsers at all, but that’s a bit outside the scope of this prototype.