The audio support in the Silverlight 2 MediaElement is limited to WMA and MP3 files. I always thought the omission of WAV file support to be a bit peculiar. After all, at some point during playback, the WMA and MP3 files are decompressed to the pure audio samples that comprise WAV files, so why not accept WAV files directly?

It's not that I wanted to play actual WAV files in a Silverlight app. No, I had something a little more ambitious in mind. I thought it might be fun to generate audio data dynamically and feed it into a MemoryStream attached to a MediaElement through the SetSource method. This would make possible an interactive electronic music synthesizer right on a web page. But the omission of WAV file support from Silverlight 2 was a big problem: I'd have to generate the audio data, and then convert it to either WMA or MP3, which involves lots of messy code and license issues.

In Silverlight 3, however, we have a solution. You still can't attach a WAV file or stream directly to a MediaElement, but you can derive from MediaStreamSource and attach it to the MediaElement and funnel either a WAV file or WAV data into that.

With those preliminaries done for me, I was able to slap together a primitive sequencer for Silverlight 3. (A sequencer is a electronic music instrument that plays a sequence of notes repetitively.) You can run it from here:

The button at the lower-left plays and pauses. Use the mouse to drag notes up and down the two staves, from two octaves below and two octaves above middle C. Press Shift and click a note to toggle between sharp, flat, and natural. Radio buttons give you a choice of simple waveforms. The top scrollbar controls the tempo; the bottom one controls the volume.

Very little testing has been done with this program. I'm not even sure the pitches are correct, and I'd be shocked if I got the sharps and flats logic right! The sounds I'm generating in this thing are just naked waveforms without any filtering or processing or envelopes. Obviously getting better-sounding sounds is high on my agenda of enhancements.

Although I am overjoyed I can now generate music in real time in Silverlight, there are some problems, in particular a very noticable latency issue. Internally, somebody (MediaElement?) wants a very comfortable buffer, so it won't start playing anything until it has that buffer, and if you change something dynamically, it won't be reflected in playback immediately. I suspect this problem will eliminate any type of interactive playing of music via the computer keyboard, but I'm curious to see how far I can get otherwise.

The source code is
here. When you derive from MediaStreamSource and attach it to a MediaElement, it's going to want data at the rate of 176,400 bytes per second (for a 44,100 sample rate, 16-bits per sample, and stereo — half that for my mono sequencer), and one way to do this is to set up a pipeline that delivers this data on demand. My MediaStreamSource derivitive requests samples from a class called Attenuator, which controls the volume, which then requests samples from the Sequencer class, which controls the note-to-note sequencing, which then requests samples from the Oscillator class, which actually generates the waveform data. All timing in the program is based on those original requests.

As I get deeper into this stuff (and if people seem to be interested), I'll discuss some of the principles of digital electronic-music synthesis in upcoming blog entries.

This looks like great fun, and I'm only getting started.

Comments:

Nice project

I'm curious to know how the interactive playing of music via computer keyboard will work out... I see that latency and jitter are an issue with flash (As3.0).

Is it possible to to write a silverlight application which allow playing virtual pairy , we have an windows based application which do the same but want online presentable solution.Can we also write virtual guitar. Is this some thing out of boundary for Silverlight.

— Jack, Thu, 23 Jul 2009 14:12:24 -0400 (EDT)

Pretty cool example. I gotta take a look at the source code.

One question about the red dot indicating the current note. With the latency of the MediaElement, how do you keep that dot accurate?

I was waiting for someone to ask me that: Look at the use of the timedPositions collection in MainPage.xaml.cs. I record the time the note begins, then wait for that time from the MediaElement. — Charles

Nice work...as usual Charles. I was browsing through your site...you are a true renaissance man.

Do you think Silverlight would be capable of making an audio mixer? Something that has...say five channels of audio with volume control for each channel, then a "master" play/pause/stop function?

I am new to Silverlight and want to try to make one...but not sure if Silverlight could do it.

— Dav, Sat, 28 Nov 2009 20:20:14 -0500

Hi Charles

This is great. I have been working on a 100% SilverLight music teaching system myself and I can now get it to play tunes synched to notation (mainly Guitar tab but I am also offering standard music notation). I can also synch it to video of the teacher playing the tune. Have a look at http://strumdiddle.com/Lessons.html

The secret for me was to include the music sound samples in the XAP package and that way I don't have any wierd timing issues. It works pretty well.

I have always loved your books and I've been reading them since your first Programming Windows way back.