Midichlorians in the blood

Sunday, April 3, 2016

Android is an operating system developed by Google around the Linux kernel. It is not like any other Linux distribution, because not only many common subsystems have been replaced by other components, but also the user interface is radically different based on Java language running into a virtual machine called Dalvik.

An example of subsystem removed from the Linux kernel is the ALSA Sequencer, which is a key piece for MIDI input/output with routing and scheduling that makes Linux comparable in capabilities to Mac OSX for musical applications (for musicians, not whistlers) and years ahead of Microsoft Windows in terms of infrastructure. Android did not offer anything comparable until Android 6 (Marshmallow).

Another subsystem from userspace Linux not included in Android is PulseAudio. Instead, OpenSL ES that can be found on Android for digital audio output and input.

But Android also has some shining components. One of them is Sonivox EAS (originally created by Sonic Network, Inc.) released under the Apache 2 license, and the MIDI Synthesizer used by my VMPK for Android application to produce noise. Funnily enough, it provided some legal fuel to Oracle in its battle against Google, because of some Java binding sources that were included in the AOSP repositories. It is not particularly outstanding in terms of audio quality, but has the ability of providing real time wavetable GM synthesis without using external soundfont files, and consumes very little resources so it may be indicated for Linux projects on small embedded devices. Let's take it to Linux, then!

So the plan is: for the next Drumstick release, there will be a Drumstick-RT backend using Sonivox EAS. The audio output part is yet undecided, but for Linux will probably be PulseAudio. In the same spirit, for Mac OSX there will be a backend leveraging the internal Apple DLS synth. These backends will be available in addition to the current FluidSynth one, which provides very good quality, but uses expensive floating point DSP calculations and requires external soundfont files.

Sunday, November 2, 2014

I've released in the past weeks some things labeled "Drumstick" and also labeled "1.0.0". What is all this about?

Drumstick is the name of a set of Qt based libraries for MIDI processing. Current major version of the Qt Frameworks is 5, which are binary incompatible with the older Qt4 libraries. Latest Qt4 based drumstick release was 0.5.0 published in 2010. Newest Qt5 based release is 1.0.0, published on August 30 2014.

Drumstick 1.0.0 is not binary compatible with the older one, nor even fully source compatible. In addition, it contains a new "drumstick-rt" library which is a cross-platform MIDI input-output abstraction. Based on Drumstick 1.0.0 I've released two more applications: vmpk 0.6.0 and kmetronome 1.0.0 (now renamed as "Drumstick Metronome").

There are other applications based on the old drumstick 0.5.0 libraries out there: kmid2 and kmidimon. I'm no longer the kmid2 maintainer, but I will release (time permitting) a "Drumstick Karaoke" application replacing kmid2, and of course also a new kmidimon (naming it as Drumstick-Whatever). Meanwhile, Linux distributions may have a problem here shipping the old and new programs together. Not a big problem, though, because the runtime libraries are intended to co-exist together on the same system. The runtime dependencies are:

vmpk-0.6.0 and kmetronome-1.0.0 depend on drumstick-1.0.0

kmidimon-0.7.5 and kmid2-2.4.0 depend on drumstick-0.5.0

If you want to distribute all kmidimon, kmid2, vmpk and kmetronome latest releases for the same system, you need to distribute also two sets of drumstick runtime libraries. This is possible because the old and new drumstick libraries have a different SONAME. What is needed is to also rename the packages accordingly.

For instance, you may name your old drumstick package as "drumstick0" and the new one "drumstick1", or append the Qt name like in "drumstick-qt4" and "drumstick-qt5", or keep the old one as plain "drumstick" and rename only the new one. Whatever makes you happier. These suggestions are for people packaging drumstick for Linux distributions. If you are compiling drumstick yourself and installing from sources, then you don't need to worry. You can use the same prefix (usually /usr/local/) without conflicts, except only one set of headers (usually the latest) can be available at the same time in your system. This also applies to the "-devel" packages from distributions.

Sunday, September 14, 2014

It has been a long time, but a week later after the release it is time to discuss some aspects of the latest version of our Virtual MIDI Piano Keyboard for desktop computers.

Where to start? an interesting point is the change of architecture with the replacement of RtMIDI by Drumstick-RT. This library is new and homegrown, part of the Drumstick family which includes Drumstick-File and Drumstick-ALSA as well. The motivation to create it was the difficulty of extending RtMIDI with other drivers different to the ones chosen by his author. This was not a problem in the past, because the RtMIDI sources were always included in the application, and any customization was possible and easy. Now, thanks to the Taliban of Linux distributions forcing the dynamic linking of RtMIDI this is simply not feasible - to the hell with the freedom of Free Software!. Throwing away the ipMIDI backend was not an option. On the other hand, with Drumstick RT is not only possible, but new backends can be compiled separately and installed in the system without recompiling neither VMPK nor Drumstick, because they are in fact plugins. By chance it also has fixed the bug reported (in 2009!) in ticket #15: LinuxSampler did not appear among ALSA connections, because LinuxSampler MIDI port has no flag providing proper characterization and RtMIDI (unlike Drumstick RT) filters out that port.

The replacement of the RtMIDI library with Drumstick-RT was a long time plan, not only for VMPK, but for Drumstick as well, that finally took place now. I hope that this shall be a foundation for features like recording/playback in the future. The only thing that maybe would be missing for some users is the jack-midi interface, but on the other hand Unix users will enjoy native OSS support, and also FluidSynth direct output on all operating systems, meaning also configurable SoundFonts: a very demanded feature for Windows users.

Another long time request finally implemented is the ability of displaying any number of keys, for instance 88 keys, instead of full octaves, starting with any arbitrary white note (ticket #39), like configuring 25 or 49 keys (depending on which device, laptop or tablet, and screen size you have). Congratulations to all the requesters and sorry for keeping you waiting for this feature so long.

Finally, the migration to Qt5 has happened. This means also replacing a dependency from Xlib to XCB, that hopefully will bring future support for wayland/whatever. The victim has been the keyboard grabbing feature, that was only working on Linux thanks to a now lost X11/Qt4 feature. I hope to bring it back in the future with a multiplatform implementation.

There are now binary packages for 32/64 bit Linux users that shall work on any modern distribution, in the form of installers packaged using the excellent BitRock InstallBuilder. That means including all the required dependencies inside the package, in the same way the libraries are included in the Windows and Mac OS X setup packages. In order to reduce the package weight, superfluous things like Jack support were excluded, because the new FluidSynth backend is intended to provide instant audio out of the box without requiring the users to search, find, ask, learn, install, try and tweak. Something that traditional Linux distributions have failed to do, in despite of their duty of integration and making the life easier to their users. I am pretty sure that many Linux distros will fail to provide VMPK native packages for this release like they did for the 0.5.x series (see Debian, Ubuntu and Fedora repositories for instance). Prove me wrong, and this kind of binary Linux packages would be deprecated.

Friday, December 27, 2013

A lot of time since the last post. Let me announce a new port of VMPK, for Android (4.x) devices. It is available in Google Play.

There are two apps, a paid version (0.5€) and a free one (gratis) with a small advertisement. It is not based on Qt, and it is not open source. It is a Java port rewritten
from scratch using the native Android MIDI synthesizer and native
Android themes. But on the other hand, is quite similar to the old N9
port, as you can see in the following screenshots, but with several additional features.

Being mostly a Java app, it includes some C code. The internal MIDI synthesizer is Android's "Sonivox EAS", which is part of AOS. The library is included in all recent Android versions, but it is not a public API, so I've compiled the library with a customized configuration and different features resulting a smaller binary and included it in the APK along with some other native code, mainly opensl_stream by Peter Brinkmann for interfacing the synth with Android's OpenSL ES audio output. An interesting aspect of the synth is the embedded GM soundfont using very small amount of memory, and not needing external data files.

Some features are: ipMIDI compatibility (MIDI OUT only) using UDP multicast and Wireless network. Accelerometer driven sliders for velocity, controllers and bender (like the N9 port), configurable number of keys and initial key, among other goodies.

Saturday, January 7, 2012

Whenever I talk to someone about the relationship between MIDI and digital audio, one of my favorite analogies is that of computer images.

A digital raster image like a JPG file contains a bitmap. It is equivalent to an MP3 file containing digital audio. Both JPG and MP3 files contain quality loss compressed data, although other formats such as BMP and WAV files can contain pictures and digital sound without compression, respectively. In both cases the files store a set of digitized values. In the case of images, the data are individual pixels or dots that represent colors of the cells in a matrix of rows and columns that divide the digitized image. In the case of sound, individual data are samples that represent moments of time which divides the digitized sound. The digitization consists in dividing alike the image or sound into small fragments, the number of which depends on the resolution we want to get and the size of the scanned original.

Another type of images is called vector graphics. They are not suitable to represent photographs, but drawings. SVG files that are used in many illustrations of Wikipedia are of this type. Instead of image fragments, they contain symbolic descriptions using coordinates of points, distances, lines, and colors... They have the advantage of scalability without loss of quality, and ease of arbitrary modification of some of its components and properties without affecting the rest. The equivalent of this technology in the world of sound is MIDI. A MIDI sequence contains timestamped messages such as notes, instrument changes, controls, etc.. Not a proper format for storing sounds recorded by a microphone, but a symbolic representation of music similar to a score.

Images are two dimensional objects, so the digitized images consist of rows and columns of elements (pixels), and the position of the elements of a drawing is characterized by a pair of numbers that represent its Cartesian coordinates. On the other hand sound recordings are one-dimensional, sound samples are taken at constant time intervals and also MIDI messages are labeled by their position in the time line.

The above similarities have implications that reflect additional parallelism. An uncompressed digitized image consisting of any single solid color takes the same amount of memory than an image of the same size representing a photograph or a complex composition of multiple colors. Similarly, a recording of silence (for example John Cage's 4'33'') takes the same amount of memory than any symphonic piece of the same duration. On the other hand, a simple vector image takes much less memory than a complex picture of the same dimensions. And a few notes MIDI sequence occupies much less memory than a complex sequence of the same duration made up of many notes or other messages.

The problems posed by digital images and sounds on stretch and reduction of dimensions are also similar. In both cases artifacts are generated, an effect known as 'aliasing', which can be offset to some extent by using 'antialiasing' filters. On the other hand, in the case of vector graphics as MIDI sequences, you can easily perform stretching and shrinking of dimensions and duration without risking artifacts or quality loss whatsoever.

Starting from a vector image, it is necessary a rendering engine to get a digital image that can be displayed on the screen or a printer. In the case of MIDI, a sequencer and a MIDI synthesizer are required to produce digital audio that can be used by an audio interface.

The programs Inkscape and Gimp, used in Linux for creating and editing vector graphics and digital images respectively, are comparable to the Adobe programs Illustrator and Photoshop. They cover different needs and audience, thriving on different niches. An example of this type of niche is the architects, who use vector graphics to design and represent buildings with Autocad or similar programs. These are not watertight compartments. Gimp can import vector graphic files, rendering them as bitmaps. Inkscape can also import a bitmap image as a drawing object. In each case, the users may choose the best tool for each task.

While it has been easy to list some essential image processing programs for Linux and other systems, to do the same exercise in the field of audio and MIDI is much more risky. The problem is that the way musicians work with computers is not homogeneous, with each musician working in a different way. For old school types the ideal work-flow is to note down musical ideas, develop drafts and refine compositions using tools that work with symbolic elements, producing as a final result a paper copy of the score. Rosegarden could be appropriate at this stage. On the other extreme, there are those who never in his life read or write a score, and whose only tools of creation (other than musical instruments) are the mixer and multi-track recorder. In this case, Ardour could be right.

The two applications mentioned above allow the use of digital audio and MIDI at the same time. In the same way as in the world of images, some applications are focused on the symbolic representation (MIDI) and others in a final product (digital audio). In each case, the use of the other technology will be subordinate. For instance, Ardour MIDI messages are aligned to the audio samples. It has even developed an API (Jack MIDI) to ensure synchronization of MIDI events to digital audio samples, subordinating MIDI to the rules of digital audio. Obviously this strategy does not fit adequately on all scenarios where MIDI is useful.

As in the imaging world, symbolic representation (MIDI) is probably better suited for design, drafting and composition. By contrast, digital audio is the dominant technology in the studio, at mixing stage and production, to obtain a finished product.