Making a multi-track recorder in Flash part 2

On Tuesday night I did a last minute ad hoc presentation to FlashBrighton to share the experiments I’ve done with the microphone capabilities of FlashPlayer 10.1. You should be able to see the recording at live.flashbrighton.org (if it’s not there, bear with us while we try to get it working). Click “media” to see the available videos.

I had a lot of fun and there were actually 85 people watching the live stream and heckling me (including Lee Brimelow and John “Flash on the Beach” Davey), which I really enjoyed, despite my feigned irritation. 🙂

I explained exactly how audio works, what it means when we see a wave form, and how that gets turned into audio that we can hear. And then I showed that a waveform gets all jumbled up when you mix sounds, but our brain can still separate it all.

Then I showed how simple it was to make a sine wave tuned to concert A, 440 Hz. And then we mixed in the C above it to make a two note chord. Here’s the code for that :

package
{import flash.display.Sprite;
import flash.events.SampleDataEvent;
import flash.media.Sound;
import flash.utils.ByteArray;
publicclass SimpleTone extends Sprite
{publicvar counter : int = 0;
publicfunction SimpleTone(){super();
varsound : Sound = newSound();
sound.addEventListener(SampleDataEvent.SAMPLE_DATA, getBytes);
sound.play();
}publicfunction getBytes(e : SampleDataEvent) : void{// we'll buffer 4096 samples into the audio datafor(var i : int = 0; i<4096; i++){
counter ++;
// some crazy maths here! But it's simple when we break it down... // We want the sine wave to oscillate 440 times a second (concert A) and Math.sin// completes a full wave in 2 x PI. So from 0 to 1 second we want to give // Math.sin a value from 0 and 440 x 2 x PI. //// And as there are 44100 samples in one second, we divide the counter by that number. //// Easy! ( or at least it would be if I could explain it better... )// var sample : Number = Math.sin(counter/44100*440*2*Math.PI)*0.5;
// now add the wave for the C above A. 261.626 is the frequency of a middle C// but we want the octave above that, so we multiply it by two. By adding this to // our sample value, we're combining the two notes.
sample+= Math.sin(counter/44100*261.626*2*2*Math.PI)*0.5;
// it's a stereo sound so we have to add the sample once for the left and once// for the right. e.data.writeFloat(sample);
e.data.writeFloat(sample);
}}}}

So then we went on to look at how we get recorded bytes from the microphone, here’s a simple microphone recorder and looper, hit the mouse to loop what you recorded.[sorry, flash content no longer available]

Don’t forget that you need FlashPlayer 10.1 for it to work!

I’ve also added some code that toggles playback so it doesn’t get too annoying if you’re reading this post 🙂 But here’s the simple version of the code :

package
{import flash.display.Sprite;
import flash.events.Event;
import flash.events.MouseEvent;
import flash.events.SampleDataEvent;
import flash.media.Microphone;
import flash.media.Sound;
import flash.media.SoundChannel;
import flash.utils.ByteArray;
publicclass SimpleMicRecorder extends Sprite
{publicvar mic : Microphone;
publicvarsound : Sound;
publicvar soundChannel : SoundChannel;
publicvar micBytes : ByteArray;
publicfunction SimpleMicRecorder(){super();
// set up the microphone
mic = Microphone.getMicrophone();
mic.setSilenceLevel(0);
mic.rate = 44;
// by adding the event listener, we're starting recording
mic.addEventListener(SampleDataEvent.SAMPLE_DATA, micBytesReceived);
// and we'll make a byte array to store our recorded bytes into
micBytes = new ByteArray();
// and when we hit the mouse, we'll stop recording and start // looping what we recorded. stage.addEventListener(MouseEvent.MOUSE_DOWN, stopRecording);
}publicfunction micBytesReceived(e : SampleDataEvent) : void{// append the data for the sound we've just recorded onto the end// of the byte array we're storing it all into
micBytes.writeBytes(e.data);
}publicfunction stopRecording(e : MouseEvent) : void{// removing the event listener from the mic in effect// stops the recording
mic.removeEventListener(SampleDataEvent.SAMPLE_DATA, micBytesReceived);
// and set the byte array position at the beginning so // that when we start reading data from it we'll get // the start of the recording
micBytes.position = 0;
// so, make the sound object and start playing it!sound = newSound();
sound.addEventListener(SampleDataEvent.SAMPLE_DATA, getBytes);
soundChannel = sound.play();
stage.removeEventListener(MouseEvent.MOUSE_DOWN, stopRecording);
}publicfunction getBytes(e : SampleDataEvent) : void{// we wanna give the sound object enough samples to play. // I find that it borks if you give it too many or too // few samples. 4096 seems a mid ground. for(var i : int = 0; i<4096; i++){// if we've run out of recorded bytes, move the pointer// on the byte array back to the beginning so that we're// now reading the bytes over again. In other words, // the sound is looping. if(micBytes.bytesAvailable == 0) micBytes.position = 0;
// read the next sample out of the mic byte arrayvar sample : Number = micBytes.readFloat();
// and write it twice into the sound data, once// for the left and once for the right. e.data.writeFloat(sample);
e.data.writeFloat(sample);
}}}}

You may notice in the video that I don’t quite get this to work. This is because I didn’t know what I was going to be doing until 6pm that night 🙂 And also because I was trying to play back bytes of data while I was still recording. This causes the pointer in the byte array to jump around and it gives very unexpected results!

So, next up I started talking about the challenges in making a multi-track recorder. The main one is that we have no accurate idea where the bytes of microphone data happen in actual time. And we also don’t really have a very accurate way to know when an audio file is playing.

Why is this a problem? Well imagine we’re making a multi-track recorder. We record one track, and then loop it. We record a new track over the top of it. The new track needs to be placed over the old track so that they’re perfectly synched together. If we don’t know when the microphone data happened in relation to the recorded audio then we can’t possibly sync them up!

This did my head in for ages. I tried different methods of finding out timing information from the sounds and the microphone data. I found some useful information :

SoundChannel.position – this is useful. As I posted earlier, it returns the position in milliseconds of the sound. For dynamically generated sounds (as in this example), I believe this is the time that the sound has been playing for.

SampleDataEvent.position – this is the event that is passed through when you’re either playing a dynamically generated sound or when you’re recording microphone input. In an earlier post I explained that this was in samples. Now I’ve checked this again, I can’t actually figure out what this is measured in. All I know is that to convert it to mils, I’m dividing by 8, which makes no sense to me. Anyone got any ideas about this? It is actually measured in samples. But I was confused because the default sample rate for the mic seemed to be 11K, not 44.1 as I assumed.

So here’s an example where I’m starting a timer, starting the mic recording, then starting the sound, and each line represents each one of those three. The red line is the timer, and that stays fixed. The next line is the mic position, extracted out from the mic’s SampleDataEvent, and divided by the sample rate to get mils, and the last is the SoundChannel position.

[sorry, flash content no longer available]

See how the three jump around? Each pixel represents a millisecond, and as even a misalignment of 30-50ms would sound weird, we clearly haven’t got the accuracy that we need. So how do you fix that? Well I worked out a system, it’s a bit hacky but it seems to kinda work. And I’ll write it up in the next post (or if you can’t wait, watch the recording from FlashBrighton).

15 Responses to Making a multi-track recorder in Flash part 2

Fantastic Seb! I have had this idea of creating ‘live’ waveforms by adding harmonics via sliders for ages and this looks like a brilliant starting point. Where will it end? Digital effects? Fourier transformations? Brilliant.

That was a brilliant session at Flash Brighton (watched over the the broadcast!)…

The only thing I can think of for the dividing by 8 question that there are 8 bits in a byte and the byte array is actually storing bits… that’s probably nonsense, but I noticed the same thing when mucking around with sound in a byte array in the past – very odd!

Great Presentation! Spent the past couple hours just tinkering around with it, I’ll spend a few more before I come it something half useful. Will you by chance be posting the multi-track example you were playing with at the end?

Really appriciate your work. But I have a question… Euhm, where can I find the source code from the live.flashbrighton.org session exactly? You said you were going to place it on your blog but I can’t find it? Even google can’t find it… Can you help me?

thanks for the reminder Slawson. I’m at FlashBelt right now but I’m doing some more audio work so maybe I’ll get to it soon I hope! Please feel free to continue with these occasional reminders, just to keep me on track! 🙂

Very cool. I’d love to see that 3rd installment. There aren’t any multitrack recorders for Android phones yet, so I thought I’d see if there was a flash one that it could be used on the Android once flash 10.1 comes to it any day now.