A standardized framework for building and including Web Audio API instrument/effect "patches"
#82

Assignees

Labels

Projects

Milestone

19 participants

Title sums it up... I'm envisioning a javascript plugin that allows the user to easily include different instruments/effects with a consistent API (e.g. Instrument objects with Play methods that take a MIDI note and duration as parameters). These "patches" would be built on top of the Web Audio API.

I'm also thinking that the plugin repo would encourage instrument/effect pull requests from developers, or perhaps create a separate project for gathering such submissions. Basically a way to have the community develop awesome web audio sounds that developers can easily include in their projects.

A few quick thoughts on this. Firstly, this sounds like a good idea - I have something very similar in Audiolet where you connect up a load of nodes in a reusable way. It's almost identical to the concept of subpatches in Max/PD, Synths in SuperCollider, etc, etc.

I also think it would be nice to try to set up a (fairly loose) standard for parameter naming and values so as far as possible you can use "patches" as drop-in replacements for each other. From my experience with SuperCollider this is often a bit of a nightmare, so for example some people have pan as -1 (left) to 1 (right), whereas others use 0 (left) to 1 (right), some Synths have a freq, others have a frequency, and so on. Obviously this is not enforceable, but it would be nice to at least have a style guide.

I like the concept of creating duck-typed aggregate nodes containing a subsystem of internal nodes, but I have a feeling that the Web Audio API guys might insist on leaving that kind of functionality to higher-level libraries. I'm inclined to agree, but the problem is that a lot of the Web Audio nodes are already somewhat high-level (like DelayNode, Oscillator, etc.) and combining these elementary nodes with other complex aggregate nodes might be useful. The solution will probably be to create a library that completely abstracts away the low-level node graph from the user, and simply creates wrapper functions for instantiating oscillators, delays, etc. I suppose we already have several high-level audio libraries, though I feel like most of these libraries create their own graph structures/nodes and only use the web audio api for raw output. Is that still how Audiolet works?

I'm also on board with the idea of creating a plug-and-play standard for web audio patches... basically I'm trying to figure out what we need first: the standard, a high-level library, or both.

Edit: I'm assuming when you were discussing duck-typed nodes, you were talking about being able to drop in aggregate nodes alongside native nodes... I was just pointing out that I don't think that's currently possible, and I'm not sure if the spec will move in a direction that allows that.

Yeah, I agree - I'd imagine that the way to go will be to have a library which wraps all of the Web Audio nodes with something which looks pretty much identical, but you can connect aggregate nodes to seamlessly.

Yes, currently Audiolet just uses the Web Audio API for output. When it looks like cross-browser support is on it's way for the Web Audio API I'll almost certainly migrate Audiolet so it works as a kind of wrapper library like you describe. I'd be interested to try migrating a small subsection over now to see how feasible this is - if I get a free couple of days I'll have a proper look at it.

Re: Matt's duck-typed edit up 2 comments - no, it's not currently possible (although you can do something somewhat similar by exposing .input and .output nodes on an aggregate node (i.e. just have extra gain nodes as connectors).

This ends up being a pretty complex design problem - because it's also related to VST plugin API, with registration etc. I'm definitely interested, but it's still a little way off before I can get to it.

Hey all, new guy here. I wanted to drop my ideas and get involved cause this is right up my alley and very similar to an idea I had not too recently.

With the direction this is headed I think it would be worth it to give a little extra push in the direction of modern DAWs (Digital Audio Workstations) and adopt a structure where you load samples or instruments to their own module, and then route them through a channel. A channel, in this case, would essentially be a list (of arbitrary size) of filter and/or effect modules, the last of which connects out to the main output (context.destination). The idea would be that you can route multiple samples through the same channel to apply the same filter chain with minimal effort. I think this would be pretty extensible as well in that you could define a simple interface for new filter/effect modules that just expose the first and last node in the sequence for connecting to peer modules.

Doing this essentially lays the groundwork for an in-browser DAW which would be crazy. This is a very abstracted idea so there might be smaller details that I'm omitting that would make this difficult... but I wanted to throw the idea out there for discussion. What do you think?

Probably worth writing a format for storring Web Audio graphs and JavaScript code to construct and de-construct them automatically; and declaring a default interface for inputs and outputs as a single unit.

Sorry, that was glib on purpose. The tough part, though, is that if you start trying to define a declarative format for Web Audio graphs, you'll end up limiting the capabilities of your format, likely to the point that it won't be anywhere near as useful as just providing a JS library to instantiate your graph.

We are studying ways to use Web Audio API inside @forresto's Meemoo and I'd like to share with you some thoughts that can go along with this "packaging inputs/outputs" idea.

Meemoo uses a standard way to transmit messages through window.postMessage between the iframe nodes that compose what we call a "Meemoo app". In this way we are thinking on how to include an external Web app (like @mattdiamond drone demo) as a module inside a Meemoo app, getting the buffers generated by that and sending them to another module, like a low pass filter, and them to the audio output.

Initially, We imagined a way to intercept buffers inside the external app and sending them through window.postMessage. Maybe it could create some sync problems because each iframe has its own clock, but it is a shot.

At the same time, we are experimenting with MIDI-like messages transmission through window.postMessage. Guys like g200kg is working on WebMIDILink and we are trying to use MIDIMessage objects based on MIDI Bridge to stay closer of the MIDI draft spec from @jussi-kalliokoski and @cwilso. It sounds like a simple way to make MIDI compatible Web apps. Here is an example of two external modules sharing MIDI messages through window.postMessage inside Meemoo. It'd be so interesting to have a standard way to also send and receive audio signals between Web Audio API apps. And maybe JSON representation of audio patches that could be loaded inside a host app (something that remembers me of VST plugins).

Hoping to contribute in this conversation adding those random experiences.

Meemoo's focus is on a visual patching environment for the web. WebAudio synths will be able to trigger animations, all instantly visually-hackable in the browser.

As of now Meemoo modules are all iframes, but in the master branch I'm implementing "native" modules (almost nailed down) which will be able access variables from other modules directly. WebMIDILink messages work fine between iframes, but there is no use trying to pass/sync audio buffers via postMessage, especially with my "native" node work.

@cwilso agreed. but i am thinking about how the effects can be maintained in a single library, ie: upload json of your graph to a server, and the effect can be built from the parsed json; in that sense i think a shared definition of the effects niiput and output interfaces would be useful.

While I like the idea of creating some kind of universal JSON patch format,
I'm not sure how well this format would handle the complexities of a
multi-layered patch... one that not only contains a number of native nodes,
but also aggregate nodes operating via closures and dynamic processes. I
can see JSON working well to encapsulate the general schema of a node
graph, but as for the inner workings of custom nodes, it might get weird.

Perhaps JSON is up to the task. I'm just worried about the serializability
of functions in JS.

Edit: After re-reading some earlier comments, I think I'm basically just reiterating @cwilso's concerns. Might as well just use minified javascript as the "universal format."

I wonder if there's any way the in-development iXMF spec might be usable or extended as part of this effort.

iXMF, which is an ongoing project of the Interactive Audio Special Interest Group (IASIG), is being designed as a means of setting up a standardized method of interconnecting interactive audio assets for use in media such as video games.

Idea is that there's a messaging format between musical instrument and synthesizer. The two are strictly decoupled to support the mobile web use-case (while Web Audio isn't widely implemented). I'm using a custom transfer format between instrument and synth, but would love to consolidate around the standard.

I'd definitely vote for the duck-thingie, with emulating the behaviour of native Web Audio nodes. I built a a test app about half a year ago where I used that approach, and it just made sense. Consider this my +1.

Sure :) pedalboard.js features some abstractions around this topic; mimicking the real world guitar effects approaches. Every "box" (pb.stomp.box subclass) includes an input buffer, an output buffer and a chain of effects. Buffers provide consistent interfaces, and enable boxes to "connect" to each other just as AudioNodes can. Parameters are adjusted by Pots, and switches are implemented as 3PDT footswitches.

I do plan to implement a package.json for managing pedal connectivity and settings; but it would only be limited to a registered set of pedals, i.e., you should know how to "read" and apply the settings. Of course, for a more generalized solution, it would be better if it pointed to a compiled js along with human-readable settings - as per previous comments, there are an infinite number of combinations you can apply AudioNodes to. An effect package could look like;

Meemoo apps have a graph json format: https://gist.github.com/3707631 with src and state instead of url and settings. I think that I could make a pedalboard.js module loader pretty easily. Do you want to try patching pedalboard.js modules with Meemoo?

It looks like Paul caught me parroting my views to the Hacker News crowd. While this thread is more geared toward an open source library solution, a cloud-hosted "Web Instruments" service along the lines of Google Web Fonts would be equally awesome. Perhaps the two could complement each other in some way.

I've been thinking lately of trying to get something similar to this going. Basically like the Twitter Bootstrap of web audio. The idea would be to expose a really simple API, like underscore.js, that just offers a whole bunch of functions for users to quickly get cool sounds going in their browser. So a quick example would be something like...

/* "au" here is the name space I was using in my notes on this idea, stands for AudioUtils (like AudioUtils.js) */
var coolDelayEffect = au.effects.simpleDelay(insert, arguments, here);
coolDelayEffect.connect(context.destination);

And what I would strive for is to have this library return AudioNodes that have just been prepared nicely for the user. This might entail some frowned-upon javascript, though. Here's an example:

au.effects.simpleDelay = function (these, are, params) {
var input = context.createGainNode(); // this node will be returned to the user after connections are made
var leftDelay = context.createDelay(); // this is part of the graph that will be hidden to the user
....
input.connect(leftDelay);
....
var output = context.createGainNode();
leftDelay.connect(output);
// connect similar nodes in this effect's graph to the output
...
input.connect = output.connect;
return input;
}

So these last two statements are the messy part. You would build this little segment of an AudioNode graph in this function call, define an input and an output node, but return just the input node. But the input node's connect() function is actually the output node's connect() function, so that when you call connect on the node returned, it preserves the integrity of the AudioNode graph built by the effect function. Does that make sense? I might have explained it poorly.

This would be the general approach to the whole library, because I wouldn't to completely abstract the audio api itself. I think a library that can be integrated seamlessly into someone's pre-existing web audio project would be really cool. Anyway, in the example above, you'd also have to attach a method to the input node that's returned which retrieves a list of nodes in the effect's hidden graph, in case the user wanted to run automations on any of the audioparams therein.

What do you guys think? Is that little bit of sticky javascript a deal-breaker? I would be happy to write a more thorough spec to better explain my ideas here as well. And this library would hopefully be community driven, so anyone can add their own effect or instrument. Maybe there would even be a little CLI tool where you can install someone's instrument patch on github to the AudioUtils.js file that you're using in your project. I'm just rambling / brainstorming now...

I just came across the @wapm (Web Audio Package Manager) project, looks like it's something @jsantell just started up. This could possibly provide a really interesting infrastructure for Web Audio projects.

Edit: Looks like I provided some inspiration for the SimpleReverb module... awesome.

@mattdiamond was gonna post here for feedback after cleaning up the service a bit in regards to a "spec". Wanted it pretty unopinionated to be usable without any framework for both effects and instruments. Originally approached it as a wat to also include html/CSS and widgets so each module can optionally have a "soft synth" UI ( like FruityEQ2 as an extreme case) but then how would that be managed with the package manager, should it even support that etc.. totally will post more info in a week or so when its more functional for feedback :)

@mattdiamond new site (hya.io) launched. Basically making a specification for an interface between an host and plugins: http://hya.io/docs.html
Still work in progress - let me know if you're interested.

me and @nick-thompson have been working on a spec for making interoperable audio components (effects, tools, soft synths, source nodes), riding on top of component.io.. check out the registry site, which contains live demos so you can try before you buy, so to speak:

I'm just getting into WebAudio and thought about creating a plugin along similar lines as to what @mattdiamond proposed - especially easily loading / switching instruments. My search led me to this thread...

I have created a toolset, jsaSound, for building sound models with the Web Audio API where all models export a standard user-level API for web developers. Check it out here:http://animatedsoundworks.com:8001/
It's all up on github, too.