Percussion Synthesis

My most recent research involves synthesizing
percussive sounds. I envision a real-time percussion
instrument that can create a variety of sounds,
including abstract, weird timbres. I am currently
looking at multiple percussion synthesis techniques
(mostly physical modeling),
picking out the sound properties that I like from
each synthesis technique, and figuring out a way to
incorporate these types of sounds into a real-time
synthesis method. The gif above
shows a 2D waveguide mesh in action after being
excited by a 2D raised sinusoid. This project is
currently in
its early stages, so stay tuned for updates.

The two following examples are 2D mesh waveguides
with 21x21 junctions, excited by an impulse, with
time-varying allpss filters at the boundaries and
convolved with (a) a noiseburst and (b) a room impulse
response.

Your browser does not support the audio element.
Your browser does not support the audio element.

To illustrate the impact of size on the mesh
waveguide synthesis,
the next two examples follow the same conditions as
above except the number of junctions has been changed
to 51x51.

Your browser does not support the audio element.
Your browser does not support the audio element.

These next four examples are finite difference
approximations for the von Karman plate model. The
parameters are set according to the material
properties of a steel plate. The finite difference
scheme used in these examples is from Stefan Bilbao's paper
"Sound Synthesis for Nonlinear Plates" (2005). In the first two examples,
the plate is given linear conditions with an inital
velocity of 20 m/s and 70 m/s, respectively. In the
last two examples, the plate is given nonlinear
conditions with 20 m/s and 70 m/s initial
velocity. Notice that under linear conditions, the two
examples sound quite similar except for amplitude
differences. Under nonlinear conditions, more
frequency components as well as a slight pitch glide appear in the 70 m/s initial
velocity example.

Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.

Using a real-time, modal synthesis technique, I
created two steel plate examples that have been
convolved with a noise burst and a room impulse response.

Your browser does not support the audio element.
Your browser does not support the audio element.

RBM Drum Sequencer

The RBM Drum Sequencer is implemented as a Pd patch
and is capable of generating variations of an input
beat pattern. The pattern variation uses a restricted
Boltzmann machine (RBM) trained on Drum and Bass rhythms
(or Drum and Bass + Brazillian-style beats). This
system can generate rhythmic variations for patterns
that have 3 instruments and intensity of each note is
included in the algorithm.

Here is an example of the audio output. You
will first hear 4 bars of the basic, input beat. The
following bars are variations created with the RBM
trained on a Drum n' Bass dataset as well as the repeat module in the drum sequencer. Feel free to download the Pd
extern, patch, and Python RBM training code below to explore this tool in
real-time.

Genetic algorithm drum machine

This project is a drum machine that is able to
generate rhythmic variations of an input beat
sequence. The drum machine is implemented as a Pure
Data (Pd) patch. The rhythmic pattern
variation is based on a canonical genetic
algorithm that is implemented as a Pd extern object
in c. While the generated rhythmic variations are
interesting and seem to fit the input beat sequence,
incorporating different output levels for the generated
percussion beats would greatly improve the
"natural-ness" of the generated variations.

Here is an example of the audio output. You
will first hear one bar of the basic, input beat. The
following bars are variations created with the genetic
algorithm as well as other modules in the drum machine
(mutes, circular shifts, and individual sample levels). Feel free to download the Pd
extern and patch below to explore this tool in
real-time.

Ghost Note VST

The Ghost Note VST is a MIDI Effect plugin
(written with JUCE and C++) that creates variations in percussive rhythmic patterns.
The user is able generate ghost notes at the
2nd, 3rd, and 4th 16th notes of a quarter note within
a beat pattern by specifying
probabilities for the appearance of those notes as
well as the volume of those notes. The beats that
appear are a repetition of the note that falls on the
quarter note beat (on the 1st 16th note of the
quarter note). The user is also able to remove notes from the
original pattern by specifying the percentage of
notes that should be removed.

Here are some examples of the audio output. You
will first hear one bar of the basic, input beat. The
following bars are variations created with the Ghost
Note VST. The last example is my favorite, so feel
free to skip the first two if you're short on time!

Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.

Downbeat estimation

During the summer of 2015, I had an amazing experience as a reseach intern in the Media Technology Lab at Gracenote. I developed a method for estimating the time of the downbeat in music. When you count the beats in 4/4 music as "1 2 3 4 1 2 3 4 ..." or 3/4 music as "1 2 3 1 2 3 ...", the downbeat is the "1". My task was to find the time at which the "1" occurs in different audio signals. I used a machine learning approach instead of a strictly DSP approach for this task. I trained my models only on the Ballroom Dataset and tested them on other songs that were not in the dataset. This work could not have been done without the help of my supervisor at Gracenote, Greg Tronel.

Here are some examples of the audio output. You will hear the test audio with a metronome. The bleep with the high frequency is the estimated downbeat. I am only posting good examples here, so if you would like to hear some bad examples, feel free to contact me:

Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.

Music pattern discovery with a Variable Markov Oracle

This is a collaborative project between Cheng-i Wang, Shlomo Dubnov, and myself. Using a Factor Oracle-like data structure, the Variable Markov Oracle, we search for repeated patterns in polyphonic music. This method is capable of discovering patterns in both audio and symbolic representations.

Physical model of the voice

This project is for integrating a
physical model of the voice with a live
saxophone player. I implemented a physical model of the
vocal tract and vocal folds that can
morph between vowel sounds. I also
developed a technique to specify the
sounding frequency of the voice
synthesis. Frequency trajectories
extracted from the acoustic saxophone
signal are used to control the pitch
of the voice. Future work involves
mapping other sonic aspects of
the saxophone sound to voice model
control parameters.

MBDelay

This is a multiband delay VST
plugin. MBDelay allows you to split a
sound into three frequency bands.
The amplitude, feedback, and delay time
for each band can be controlled
separately. I am working on getting this
to work on Mac OSX 10.10, but I
have been experiencing issues with Apple vecLib FFT
functions since updating my
OS. If you
have any suggestions, please send them
my way.

GrannyGrains

This is a granular synthesis VST
plugin. With GrannyGrains, you can
chunk up your audio into grains (large
or small) and play them either
sequentially or in a shuffled
manner. You can control the binaural
spread of the grains, the feedback, and
percentage of reversed grains. There is
also an option to quantize the timing of
the grains to the beat of your DAW.

Here's an example of the audio
output. The first few seconds are of a
clean music box signal. I manipulate
parameters on GrannyGrains for the rest
of the sound file.

Perceptual grouping concatenative synthesis

This was an experiment to see if you
can create interesting music using a new
technique of concatenative synthesis. A user sorts small 10-second snippets
of audio ('source') into groups. The groups are
purely user-defined. The system uses
decision trees to understand the
structure of the grouping. The user
inputs a 'target' audio for what they
want the final output audio to sound
like. The system then takes the small
snippets of source audio that sound the
most 'similar' to each chunk of the
target audio, concatenates them
together, and outputs the sound.

Here are some examples of the audio output:

Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.