AVSynthesis: Blending Light and Sound with OpenGL and Csound5

Introducing a unique and powerful program for mixing son et lumière into fascinating experimental videos.

The artistic combination of sound and image is a common enough
phenomenon. Movies, television and various Internet channels demonstrate
the happy results from the blend of recorded sight and sound. However,
these examples typically utilize sound in the role of an accompanist,
perhaps greatly significant but still primarily an accompanist.

There is another way to consider the role of music and sound in video
production—a way in which the sound itself informs the flow of images
and their transformations. Although not a novel concept (see the Wikipedia
entry on John Whitney), the practice has taken on a new richness of
possibilities with the use of computers in the recording and editing of
digital son et lumière.

Jean-Pierre Lemoine has been exploring these new riches at least since the
late 1990s. I profiled his HPKComposer (coauthored with Didiel Debril)
in my Book of Linux Music & Sound, which was written in 1999,
and even then the HPKComposer Web page stated that the program was
“... a 3D art composition tool for Csound”. At that time, the authors
chose to use the Virtual Reality Modeling Language (VRML) for its
graphics engine. I could meet the program's Java requirements and work
with its Csound side, but I was unable to work with VRML under Linux
then. Nevertheless, the Web site's screenshots made quite an impression,
and I hoped that someday such a program would become useful under Linux.

Cut to the work of Csound developer Gabriel Maldonado: his CsoundAV
for Windows is a true fork from the canonical Csound source tree, but
Gabe is a genial fellow who freely offers all his code extensions to the
community. Recent developments in canonical Csound have facilitated the
adoption of some CsoundAV opcodes, though we await the inclusion of the
CsoundAV opcodes for OpenGL, and this situation brings us to the latest
work of Jean-Pierre Lemoine, titled simply AVSynthesis (Figure 1).

Figure 1. AVSynthesis in Play

AVSynthesis embraces and extends many of the design concepts behind
HPKComposer. The program blends sound and images to produce abstract
non-representational works of art. It's written in Java, and Csound is still
the audio engine of choice, but the VRML interface has been replaced by
a set of image controls based on the OpenGL shading language (GLSL).
The
program creates radical associations and correspondences between image
and sound, leveraging the powers of Csound and OpenGL for the arbitrary
manipulation of digital audio and digital images.

Requirements and Installations

Like many experimental applications, AVSynthesis is not a perfectly
packaged program, and it is not ready for use right out of the box. It
is a unique program, and as such, it has some unique requirements that
may not be met by your distribution's package repositories. Building the
required dependencies is not especially difficult, as long as you have a
typical Linux development environment installed and properly configured
for your system. I include here the particular instructions for compiling
Csound and configuring AVSynthesis, with some notes on the requirements
for building the application on a 64-bit system.

AVSynthesis demands a specific set of dependencies:

Java (1.5 or higher)

LWJGL (the Light Weight Java Game Library)

Csound (5.07 or higher)

OpenGL

Where they are noted, the versions are critical, and each component is
subject to its own build prerequisites. As mentioned, Csound needs some
special attention in order to use it with AVSynthesis.

Csound has its own set of necessary dependencies, but space
restrictions here forbid a complete description of the program and
its requirements. Fortunately, thorough and excellent documentation
is available from www.csounds.com, so I focus here only on the
configuration needed to compile the program for use with AVSynthesis.

The following options configure and compile the csound binary for
double-precision floating-point numerics and create lib_jcsound.so, a Java
“wrapper” library for Csound's audio synthesis and processing
services:

The Python opcodes are not required by AVSynthesis, but I include the
option for use with Steven Yi's blue, a superb environment for
working with Csound. All other options in this build configuration must
be included for work with AVSynthesis. If the build is successful, the
lib_jcsound.so library will be at the top level of the Csound source
tree. Install Csound (scons install), then copy lib_jcsound.so to the
AVSynthesis native directory. That's it; you're finished with setting
up the audio side of AVSynthesis.

The OpenGL and LWJGL libraries provide the interface's visual components
and style. The various parameter control screens resemble the control
panels seen in many OpenGL-based games, with visual effects, such as
animated icons and mobile transparencies—niceties that liven the
appearance of the program and improve its work flow.

The LWJGL libraries present a minor difficulty. The AVSynthesis package
includes the LWJGL libraries as Windows-format DLLs but not the required
native Linux libraries (that is, in .so format). The package includes these
DLLs:

DevIL.dll

ILU.dll

ILUT.dll

_jcsound.dll

lwjgl-devil.dll

lwjgl.dll

Those files must be replaced by the following native Linux equivalents:

libIL.so

libILU.so

libILUT.so

lib_jcsound.so

liblwjgl-devil.so

liblwjgl.so

The lib_jcsound.so library comes from the Csound build described
above; the others come from the LWJGL binary package (downloaded from
lwjgl.org). Alas, 64-bit users will need to build and install the LWJGL
and the IL libraries themselves. As far as I could tell, packages for
these libraries are not readily available in 64-bit format, but building
them is trivial and requires no special instructions beyond adding
--with-pic to the configuration step (./config
--with-pic). After building
or downloading the libraries, they must be copied to the AVSynthesis
native directory. You then can move or delete the DLL versions.

Neither Java nor OpenGL requires any rebuilding or special runtime
options. These are common packages now, so if you don't have them
installed already, summon your package manager and install the latest
versions (Java must be 1.5 or higher). AVSynthesis itself is launched from
a .jar file that works equally well in a 32-bit or 64-bit environment.

In addition to these software requirements, your computer should have
a fast CPU and a video system capable of accelerated 3-D graphics. I
tested AVSynthesis on two machines: a 32-bit box with an AMD64 3800+
CPU (a 2.4GHz chip) and a 64-bit machine powered by an AMD64 3200+ CPU
(2GHz). Both systems include NVIDIA graphics boards (GeForce 7300GS and
GeForce 7600GS, respectively), with xorg.conf configured for NVIDIA's
proprietary nvidia driver (that is, not the open-source nv
module). The 32-bit iron runs the JAD distribution, based on OpenSUSE
10.2, and my 64-bit box runs 64 Studio, a Debian-based distro. Both
systems are optimized for multimedia and include kernels optimized
for real-time performance. However, programs such as AVSynthesis want
resources, lots of them, and I consider my machines as rather low-end
for AVSynthesis. Your mileage may vary, of course, but for the best
results from this program, I recommend a 3GHz CPU, at least 2GB of RAM,
a fast 3-D graphics card and a large, fast hard disk.

I also recommend a high-quality audio system. Cheaper desktop speaker
arrays may be suitable for watching DVDs, but Csound is capable of
audiophile-quality output, so you'll want a sound system as powerful as
your graphics system. Here at Studio Dave, I have my JAD box connected
to a relatively low-end 5.1 sound system (a combination of Creative
Labs and Peavey hardware), while the 64 Studio machine is hooked up to
a conventional small studio audio system with a Yamaha digital mixer,
a standalone 100-watt power amplifier and a pair of high-quality
monitor speakers.

AVS has gone through a series of updates since I wrote this article. Please see the AVS Web page for more information. Significant improvements include randomization controls, a version for JOGL (intended to replace the LWGL dependencies), and better image-size support.