Stefan Westerfeld
has posted
a summary of a IRC discussion held by the KDE Multimedia team
last month. Essentially a KDE 3 roadmap for the multimedia team, the
discussion covers topics ranging from MCOP and OSS compatability to recording
and video embedding. A slightly edited version of his post follows.

This summary provides an overview of the things that we talked about
during the KDE3.0 multimedia meeting which took place on IRC (2001-09-04).

We talked our way through more or less the topics that were posted before on
the list. I will summarize this document to a list list of keywords for
the KDE3 planned features document (which is relevant for the freeze), but
this is the more verbose source of what will be done.

MCOP<->DCOP bridging: the plan is to provide access to MCOP based
services from DCOP - thats useful for scripting, as well as development,
and desktop integration.

==> responsible: Nikolas Zimmermann

JMCOP: There is a working but unreleased partial Java MCOP implementation,
so if you want to work on something like this, don't start from scratch -
however, there is currently no fixed completion date for this, nor will
it probably be distributed bundled with KDE3.0.

UCOM: There was a bit discussion about this one, but nobody seemed to
have looked too much into UCOM yet.

Threaded OSS support: some applications acquire better results with
respect to compatibility with various possible kinds of hardware than
aRts, because of their threaded implementation of accessing the sound
card. The threaded oss patch is a patch that opens the driver in blocking
mode and spawns a seperate thread.

While there was discussion whether the observed increase in quality (less
dropouts, better support on more hardware) was caused

additional buffering gained through this (especially for cards with
small hardware buffer)

brokenness of various OSS drivers for completely non-blocking I/O

the usage of select(...) which might be badly implemented for some
OSS drivers or a combination of those, it generally was considered a good
idea to do it

==> responsible: Matthias Welwarsky
[ I wrote him here because he wrote the patch, but since he wasn't
at the meeting, we might need to change this ]

All strings emitted by aRts should be internationalizable in KDE3.0 - they
are currently not, because aRts does neither use libkdecore (so it can't
use the standard i18n), nor has own means to deal with that.

While workarounds might be possible (such as artsd only emitting error
codes, and seperate in-the-gui-layer-translation), the right solution,
which was also somewhat discussed on the lists will be adding own i18n
capabilities to libmcop. Whether or not libc i18n can be used, or whether
the kdecore code could be copied, is still to be evaluated. As there are
no unicode strings in aRts (no QString), utf8 is probably the way to go
for transport to the GUI.

The aRts C API didn't implement clientside recording until now, but it
should. Likewise should the Arts::SoundServer interface. It might be good
finally following the KDE2 plan to provide a KAudioStream class in
libartskde, to provide recording/playing streams in a nice, canned,
easy-to-use way. Including an end user sound recorder into kdemultimedia
might be useful, too.

There is an VideoPlayObject interface in KDE2.2 already, which supports
playing videos via X11 embedding. This is a quite simple solution, but
using it more in KDE applications will make viewing videos much more
user friendly than it is now.

There was some discussion about videos and noatun. Noatun currently doesn't
use the embedding interface, and this might be hard to do, because every
skin needs to be changed for that. On the other hand, noatun is the
official KDE media player, so it should probably be able to embed videos.

There was some discussion about eventually implementing video editing, but
we currently lack mainly the codecs to do it. The codecs don't support
precise frame based seeking and delivery of the raw image as data. Besides
this, video support would greatly benefit from introducing more structured
asynchronous data delivery (with timestamps and frames) into aRts.

Extending Brahms towards embedding on the other hand seems less hard, so
that it might be reasonable simple to compose film sound tracks with Brahms
eventually, or - if we have better codecs - even edit films the same way
Brahms can edit music.

Malte does have a RealMedia PlayObject ready (based on the Real library, so
the implementation is not entierly open source). Currently, there are
unsolved licensing issues.

==> responsible: Malte Starostik

There was some discussions on other ways to get codecs. Basically, writing
a PlayObject for one of

ffmpeg

libmpeg3

xine

gstreamer

could extend the fileformats aRts will be able to play. Martin Vogt is
working on xine, combined with code to standarize rendering and output
more, to make integration of new video codecs, and later, video editing,
simpler. There is experimental libmpeg3 code in the CVS.

Midi sequencers (Brahms/Rosegarden/Anthem) have started to use aRts
as way of outputting midi, whereas other applications (kmid) still use
libkmid, and some things (for instance ALSA) are not supported by aRts
at this time. For KDE3.0, aRts should provide a complete solution.

The midi interfaces should finally support accessing sound card hardware of
all fashions (ALSA, OSS-hw-synth) - that will mean a rewrite of some stuff,
but is necessary for making the midi interfaces reasonable complete.

Precise synchronization between midi/audio should be adressed.

Libkmid could finally be moved upon the new midi interfaces.

The midi libs should be mainstream (i.e. kdelibs) and stable after KDE3.

A midi PlayObject would be helpful, to enable media players like noatun
to deal with midis.

it allows putting module calculations in a seperate thread from the
aRts main thread, thereby avoiding dropouts from lengthy aRts operations,
such as for instance calling open() on an NFS drive

it is designed in a way to support advanced scheduling features that
aRts currently does not support, that is

feedback loops (circles)

sample precise timing

it should allow achieving very very low latencies reliable (where very
very low is something like 3ms) - aRts currently can't do this

it is implemented in plain C, using some glibisms, however it can be
integrated into aRts with a glib-compat source in a way that it does
not bring additional dependancies (no glib required)

The GSL engine should first act as a drop-in replacement - however to take
advantage of its capabilities, it will be necessary to make sure that the
modules are implemented in a way that they can run in a seperate thread
without interfering with anything else.

That means, that there will two types of modules: the old modules (which
will still be running in the aRts main thread), and the hard-rt-modules
(which will be able to run in the engine threaded).

You will get the very very low latency benefit only if you use only hard-
rt-modules, but everything should work with any mix of modules.

Where is GSL from?

GSL is designed by Tim Janik and me, and contains a lot of know how about
music apps (Tim Janik has been working for years on Beast - best.gtk.org,
and I have been working for years on aRts).

GSL code is in the CVS and working (although yet somewhat slow) already.

GSL provides a sample cache which supports partial caching and preloading,
the integration of that will get rid of the long outstanding issue that
aRts always needs to load whole WAV files (or other formats), and support
GigaSampler like "read-from-HD-and-play" style sample sets of huge size.

GSL provides filter design code for butterworth, tschebyscheff-1/2
and soon elliptic low/high and soon bandpass/stop filters, which can be
wrapped in new aRts modules (aka Arts::Synth_HIGHPASS).

While aRts is good at creating synthetic sounds and using these as
instruments in a sequencer (i.e. Brahms), playing samples will greatly
enhance the possibilities.

I have a prototype of a .PAT loader - you can import a whole timidity
style patchset with that, which means that we have a useful GM-sample-set
for aRts. There is also AKAI loading code in kmusic/ksynth. GSL provides
another native format. All of these probably should use the GSL sample
cache for efficient, preloaded, replaying.

Combined with an aRts PlayObject, media players (noatun, kaboodle) gain
not only the capability to do similar replay than kmid (output midi via
card synth or external midi), but also kmidi (output midi as calculated
audio data from sample sets).

The work started with mixers/environments should be completed - for instanceit would be very useful if the mixer could save/restore itself properly, and
if it could also contain insert-stereoeffects and a master volume. Also,
the mixer look more like a real-mixer(tm), for instance groups or panning
is not there right now.

==> responsible: Matthias Kretz

Visualizations (led-bars) would be nice, but currently have the technical
problem that they overload the MCOP layer easily, this can probably only
be fixed by changing MCOP to deal more intelligently with change
notifications.

There are about three major areas which the work focuses on. First of all,
it's important to continue to improve everyday usage for casual users. Mostly,
recording, i18n and the threaded OSS support fall in that category.

On the other hand, a lot of things should be simply unified, made more
consistent, and merged. A media player which can play everything might sound
a lot better to the user than noatun, kaboodle, kmid, kmidi and aktion.

Finally, there is a lot of work to be done especially for music applications,
which includes the gsl engine, more support for samples, midi improvements,
and environments/mixers.

I hope that with the way we split up the work/responsibility on several
developers, we'll be able to achieve the goals listed here in time for
KDE3.0.

Comments

I'd like to see someone throw together a KDE front end to the ogle DVD player. It has a backend and a gtk front end so a KDE interface would be cool. If you hav'nt tried it you should. Its much more simpler than xine and actually works. It does menus and encryted dvd's as well. All i had to do was make a symbolic link from /dev/dvd to /dev/cdrom it get it to work.

I love Free Software, but I do not have the free time to help out on any projects. If I were to write something for someone, I would have to charge for it, or they would have to wait a *very very* long time for it to get done. Perhaps costing them more time/aggrevation than just paying me.

Free Software is wonderful, but I don't see anything wrong with someone getting paid, nor asking/wanting to get paid for his work. A workman is worthy of his hire. If he wants to work for free, fine, if he doesn't/can't that's fine too. I can't.

I agree that noatun needs seriously implementation for things like dvd / divx / avi files (the ones aktion can read through xanim) / good mpeg support.
You might say that there s a div x plug in already, but

first you need to download it seperately (okay, not too bad)

second it has a lot of dependancies, and it takes some time to figure out which libs are needed and which are not. (!hint: the plug in page should contain direct links to all files needed)

last point I never made it work (which may be my fault

avi support shouldnt be to heard to be ripped from aktion and made ready for noatun

mpeg okay, the codec works quite well. Anyone cosidered *resizing* a window, or full screen?

why create redundancy? because Mplayer doesnt use QT, therefore its wrong... man you guys need to get over the QT for everything attitude. we have enough media playing apps in linux, most of which are gtk+ based. why not just use this?

mplayer works and performs fine, why waste your time trying to QT-ISE it?

this is like re-creating a winamp-type mp3 player, whats wrong with xmms? oh yeah thats right..its not QT.. you guys are just pathetic.

I think there's nothing wrong with creating a beautiful KDE
interface and *KDE integration* of mplayer. I must say that
mplayer is a **very** good piece of software for the user that
uses it. I don't know about software engineering techniques.
Maybe you should use the mplayer kernel in the NOATUN
environment. Though of course it is largely i386 based. Or am
I wrong here...

a) most distro have gtk+/glib installed by default
b) many apps use gtk, like gnumeric, xmms, gimp, abiword, evolution, mplayer.

hey even the most hard-core kde users will still use some gtk apps,...just keep on denying it..if it makes you feel better.

c) whats the worse it can do? take up a few mb's?

consider this, gtk+ is owned by the "community"...QT is owned by a company, i can see why i prefer this. sure it has investment from a few companys, but hey its not fully controlled by any of them.

Trolltech own you, they make the decisions... the GPL license of QT on the linux side of things is just a cover, considering that OS-X and windows dont have a GPL version. Probably since users of both platforms arent screaming to trolltech about no GPL/QT, thats the reason why. they own you.

konq is such a piece of broken shit btw, you guys are trying to hard to be IE.. ditch khtml, mozilla is so more usable it makes konq look like a joke. btw why does KDE act so much like windows? do you guys have any sense of innovation? I choose linux to get away from windows...not see another windows-wannabe.

why not ditch the QT/GPL license and just place it under LGPL? GPL is shit license for libs.

.. you're obviously not here to add anything constructive to this conversation, just to cause trouble. This site is about KDE, for discussion of KDE subjects and the KDE software by KDE users ; so if you want to wave your willy about gtk, take it elsewhere.

he lost all claim to validity when he started using 4 letter expletives to throw insults at the products of other people's hard work and generously given free time; because they doen't happen to match HIS tiny world view. I suggest you scan all of his posts here, he's just trying to start a fight.

They'd want to "waste time" QT-ising it so that it becomes part of the desktop, i.e. so you can integrate it as a KPart, and control it with DCOP, and have it use the same theme and colors as the other apps. GTK based apps have basically no access to the infastructure that makes a KDE app part of the desktop, similar to how QT apps have little or no access to GNOME infastructure.

Why i whould have to load GTK? i already have QT and KDE libraries loaded,
why should i waste more memory with another widget set, most of you don't care
but i do have little memory and i want to keep it's use as little as posible
to have more resource to other importan apps like OpenOffice or other things
i use. Though i don't see why not use gmplayer, generally you'll use only
mplayer to see movies instead of OpenOffice and Mplayer with xmms together.

I always waited for these features!
So now we get a fine multimedia player but most notably we get a good basic framework for software like (usable) music sequencers etc. which is the only thing I miss on linux.

I hope Brahms or other software like anthem and noteedit implement those features and will be stable (!) shortly after the KDE 3.0 release. The last time I compiled it (which btw was the first time I managed to compile it with KDE 2.x) it seemed to be still missing many features and options.

I always waited for these features!
So now we get a fine multimedia player but most notably we get a good basic framework for software like (usable) music sequencers etc. which is the only thing I miss on linux.

I hope Brahms or other software like anthem and noteedit implement those features and will be stable (!) shortly after the KDE 3.0 release. The last time I compiled it (which btw was the first time I managed to compile it with KDE 2.x) it seemed to be still missing many features and options.

I always waited for these features!
So now we get a fine multimedia player but most notably we get a good basic framework for software like (usable) music sequencers etc. which is the only thing I miss on linux.

I hope Brahms or other software like anthem and noteedit implement those features and will be stable (!) shortly after the KDE 3.0 release. The last time I compiled it (which btw was the first time I managed to compile it with KDE 2.x) it seemed to be still missing many features and options.

Has there been any consideration of integrating XMMS more tightly with KDE? It seems to be a much more usable player than noatun for dealing with music due to its familiar (and likely superior) interface. Even if you guys end up creating a program similar to Media Player on win32, it would be nice to have the choice to keep XMMS as an mp3 player (this of course can still be done, it just won't be as integrated into the KDE multimedia layer as the KDE player would be). I don't know much about the implementation of XMMS or KDE's media players, but it seems like it would be less work to integrate XMMS into KDE than to duplicate its functionality in KDE. I just think that some people still like to have a separate mp3 player instead of an integrated one, and that XMMS fits the bill well.

The integrated player will also be a very welcome addition to KDE, and it is probably of more importance to get good video capabilities in one program now than anything else. I'm just wondering whether something like XMMS could still have a place in KDE along with the new media player(s).

You also have to think of this from a developers point of view. When you are creating a KDE application it would be much easier to use the KDE media libraries rather than using pieces of XMMS / another media player.

aRts is independent of KDE. They're trying to stay away from it as much as possible AFAIK (hence the use of MCOP and a MCOP->DCOP bridge). It runs just as well under Gnome, and in fact is supported by the Gnome Control Panel.

Basically it's attempting to usurp ESD's role, while bunging in a whole lot of high level functionality as well.

Why not create an ability to use XMMS input plugins. There's a whole raft of them about, and they seem to be seeing a lot more active development than the KDE equivalents. There's no point re-inventing a wheel that good.

Personally, I love KDE, but I've never found noatun/kaiman to be acceptable. XMMS is what I use for almost all my video and music.

That sounds nice, but it has complications. Integrating XMMS input plugins would make them available to Noatun, sure, but if we do it as aRts plugins and put everything as low level as possible, then you will be able to later, say, put all the audio and video formats that Noatun supports into usage inside KPresenter presentations, and previewing inside Kaboodle, and lots of other things I'm sure I haven't thought of yet.

I looked into that once, but at least when I looked at it, XMMS input plugins had two disadvantages.

1. They are usually designed to play/decode only one file at a time, which makes them unsuitable for a media framework, where you can't know whether a DJ application will be using your playing components (and thus, you will need to play two mp3s at a time for crossfading).

2. They tightly integrate with the GUI for configurations. Thus, you can't do the configuration in the client application, while letting the actual decoding take place on the server (which is what apps like noatun need to do). You actually even need a Gtk+ event loop to use the GUI dialogs, which you can't provide inside artsd.

XMMS is a great player, and there is no reason why XMMS fans shouldn't use it, together with the XMMS-aRts output plugin in KDE. But in my opinion, its code is largely unusable for anything but a player, whereas aRts is trying to be fairly generic.

> Has there been any consideration of integrating XMMS more tightly with KDE? It seems to be a much more usable player than Noatun for dealing with music due to its familiar (and likely superior) interface.

I use XMMS because Noatun has some weaks, but I would prefer using noatun. So I hope that in KDE 3.0, it would acquire the functionnalities of XMMS.

For me, now, the biggest lack is that clicking on a m3u link in Konqueror does not load the content of the m3u in the noatun playlist. Very bad integration ! (perhaps is it a problem of Konqueror more than Noatun ??... I don't think, because it is good with XMMS...)

try to google for beep... if I remember correctly it should use gtk2 which can be KDE-ised with the qt-gtk-theme(this is especially worth dowloading! http://www.freedesktop.com)
pretty unstable now but under development, they also want to support winamp3 skins -> whoa!!!

Well, on FreeBSD noatun does not work at all. xmms does. Dunno, wether this is KDE (arts) or port specific. Same for mplayer: the only Videolpayer that works here (well, vlc does fine for mpeg).
But the need of disabling arts to be able to listen to music is not really cool.

Assuming that the port maintainers of kde or I myself are not dumb, maybe a bit more portability would not really mind. In case the latter one is true, just forget about this post.

Finally - under Linux - I have trouble when having two or more KDE sessions open with diferent users. Would be nice if both were able to play music since I thought that is what arts is all about - playing multiple streams simultaneously. Even when different users try to do this.

Most of the solutions for playing video uses win32 dlls. This way of doing things just don't work very well (to talk the truth, on my system it's asking for a total freeze - I think the guilt if from xfree 4.1...).
Anyway, I ask you to please try to do a native solution. I know using winelib/dlls is a lot easier, etc and tal. But we need native solutions! Can't ypu reverse eng. the codecs or something like it?

Most of the solutions for playing video uses win32 dlls. This way of doing things just don't work very well (to talk the truth, on my system it's asking for a total freeze - I think the guilt if from xfree 4.1...).
Anyway, I ask you to please try to do a native solution. I know using winelib/dlls is a lot easier, etc and tal. But we need native solutions! Can't ypu reverse eng. the codecs or something like it?
Oh yeah, and noatun dosen't work for me, I vote for a qt/kde xmms :)

I guess I'm the only person that doesnt like arts out there. I have tried and tried with it, but no matter what I do, it always ends up crashing a few minutes after I get it going. sometimes it makes it through a few songs, but then, right as i begin to think, maybe it's working, it craps out on me again. i use slackware linux and have tried using the packages that slackware-current has out, and compiling everything myself. even when i tried using artsd on a different machine running red crap 7.1 it still crashed on me after a few minutes. i think the best thing KDE could do for the entire project would be to switch to esd, which has never ONCE failed on me on any computer for any reason.

Umm, if aRts is failing on your system, then it's a problem with your setup. aRts does not normally behave that way (if it did, do you think KDE would be using it?). ESD is also considerably less powerful then aRts, from the perspective of powerful multimedia apps like MIDI editors (Brahms uses aRts's own functions for traversing and altering MIDI, iirc)

Usually thats a problem of the kernel driver you are using. A lot of linux applications (like xmms/esd) use only a part of the kernel driver API, namely blocking I/O. Artsd uses a different part, namely non-blocking I/O. Which is the reason why some kernel drivers (such as the aureal driver, or i810) have problems running artsd, but no problems xmms/esd. So the right place to complain about it is the author of the driver.

See also the "Compatibility with legacy/broken OSS drivers" in the KDE3.0 planning document.

You're not the only one. I hate it because it takes over my sound card. I'd like to be able to run my kde sounds through arts, but still be able to put other apps like avifile through the direct /dev/dsp.

You're not the only one. I hate it because it takes over my sound card. I'd like to be able to run my kde sounds through arts, but still be able to put other apps like avifile through the direct /dev/dsp.