I have an idea for an LV2 plugin, that'll synthesize sounds based on input images, treating them as spectrograms.Basically, the reverse of generating a spectrogram, in a DAW-friendly way, that'd also allow the users to play musical notes (pitching the synthesized result according to MIDI input).

That'd be an image-based synthesizer. I'd rather thing of this a s a tool for sound effects design, but coudl als obe used in music.

Right now the only tool I've found in the Linux audio world that can synthesize sound from images is ARSS, and it's a bit of a pain in the ARSS to use it

To synthesize a sound from an image you have to:

1. Download ARSS binary (https://sourceforge.net/projects/arss/)2. Prepare the image3. Save your image it as a specific BMP format, becasue it's the only one supported by ARSS4. Compose a shell command to synthesize a WAV file from your BMP file5. Run ARSS6. Listen7. Tweak the commandline parameters and repeat from 4. , or tweak the image and repeat from 2.8. You now have to use a sampler plugin or another way of putting that WAV file into your DAW session.

I wish there was a DAW-friendly way of doing this, likeusing an LV2 plugin that can just load an image file, have some knobs I can tweak live. It'd be fantastic, if the sound synthesis could happen in realtime, because that'd make it possible to automate some parameters, and have that actually reflected in the sound.

ZynAddSubFX's PADsynth generates samples under the hood (and it works as an LV2 plugin) so I guess that's one approach. I think that Image-Line Harmor synthesizes sound from images on the fly (and does a lot of other addtive synthesis awesomness, but that's not what I'm looking for).

It is real-time, fully open source (JavaScript client/server side/advanced sound server in C), you can import (drag & drop) images or sounds and display it as you want with few lines of live GLSL code, play it forward/backward/any ways from a MIDI controller (MPE ready) or from code, the played pixels data can also be recorded, imported back and played again, since it use the GPU you can do whatever to the pixels data in real-time, i suggest to watch all the videos for all the possibilities.

The audio to images conversion is not as good as ARSS though, the sound engine may show off some limits as well because it use a (albeit very large and basically unlimited) heavily optimized bank of oscillators, the phase data is accessible but not used, using FFT/IFFT is planned.

There is also an Audio Server written in C which is quite fast and is able to add band-limited noise, this was made as band-aid performance and because i wanted to output the sound to my DAW of choice you can feed it pixels data and it will output sounds synthesized with additive or granular synthesis, the program is quite generic, has simple protocol and is easy to compile and can be found here : https://github.com/grz0zrg/fas

One of the advantage of the sound server with the fact it has many settings and support granular synthesis, is that it support distributed sound synthesis out of the box, many cores can be used (memory hungry) or even multiple machines over the network, someone could literally build a beast of a synth with it with an almost unlimited partials range.

This can be used in a DAW (it support multiple output channels) although it is not as DAW friendly as many would want, basically you also need a client (like Fragment) to feed it the pixels data through the network via WebSocket.

Anyway i am actually writing a DAW-friendly software with a bit of an unique twist not yet seen from all the software that do that already, i don't have any knowledge of plugins though so i am rather slow at making it, but if anyone want to team up...