Update: This project was dead, but it's back! Unfortunately my daytime job keeps me somewhat busy, so don't expect the project to emerge in a couple of years.
Saya is a programming project aimed to create a cross-platform, versatile, extensible Nonlinear video editor with features available in commercial editors like Adobe Premiere, Sony Vegas or Edius Pro. The project was started in June, 2008, died in 2009 and was revived in October 2010.

Monday, June 23, 2008

Recently I've been writing on a mailing list about the ffmpeg project, and on why we need an alternative. A little bit of browsing led me to take a second look at Gstreamer. Therefore, my previous post entitled "On Diva, gstreamer and tiers gone wrong" seems to be inaccurate.

In theory, GStreamer is everything a video editor has dreamed of... but I couldn't get hold of the developer guide (it's on development. How ironic). However, I have this hunch, telling me that I won't be able to access frames and audio samples directly, and instead I'll have to rely on plugins such as gnonlin.

Let's hope not. I mailed one of the Pitivi authors for guidance. Let's see what we find out. Cross your fingers.

Saturday, June 21, 2008

Perhaps this will help other developers who want to write cross-platform applications with Code::Blocks. After breaking my head for various hours, I finally found out how to do it. Code::Blocks 8.02 includes a script engine which can make certain compiler options apply only to Windows environments.

(Note: The double ;; at the end is a workaround a scripting bug)As you can see, the one-line script is pretty much like C. the _T must have been incorporated for wxWidgets compatibility. In any case, Notice the compilation string: -mthreads, -D__GNUWIN32__ and -D__WXMSW__. The -D is a compiler define. I tried to add scripting to the #defines, but that didn't work, so I had to add them on the compiler command line.

The wx-config part is for Linux. I've found that experimentally, having this option in the project doesn't affect compilation under Windows.

The same can be done with the linker. Under Linker settings, other linker options:

Here I tell the linker to link two windows-only libraries: libwxmsw28u.a, and libintl.dll.a (libintl is used for internationalization). For posix environments (OS X or GNU/Linux), the backticked expressions are more than enough, but you could do the same with "if (PLATFORM != PLATFORM_MSW).

Under search directories, I add both the Windows and posix directories.

Compiler search dirs:

$(#wx.include)$(#wx)/contrib/include$(#wx)/lib/gcc_dll/mswu

Linker search dirs:

$(#wx)/lib/gcc_dll

Resource Compiler search dirs:

$(#wx.include)$(#wx)/lib/gcc_dll/mswu

With this simple settings, you won't require having two different projects, one for GNU/Linux or Mac OS, and another for Windows.

Friday, June 20, 2008

Today was a hectic day. Not only I had to run Windows to get my tax declaration working (the app doesn't run on GNU/Linux), but I had to use Windows to set up Code::Blocks as well.

Once upon a time... the second most active developer, Nopalin, had problems with his harddrive and ended up installing Windows. The pandora box had just been opened on me. I had now to download MINGW, compile wxWidgets, download an additional libintl from gnuwin32 (it doesn't come by default with mingw), install subversion, and guess what.

As I had expected, I had compiling problems with the project, which was configured for a posix environment. I had to modify the global settings for Code::Blocks under Windows (something you must NOT do, that's what the project settings are for!) until I found out how to make the project build properties cross-platform (this is a capability not implemented in Code::Blocks yet, maybe they have scripting capabilities now, but I don't know of them).

And then wxDateTime method FormatISODate() outputs garbage on Windows. When that was fixed, I noticed some other problems: The buttons in the welcome dialog were invisible, and the new project dialog has the wrong size! :(

How did all this happen? Well, I'll be able to find out as soon as I make the project cross-platform. In the worst cases, I'll have to use the same tactic the C::B devs did: Use parallel project files. Ugh. Let's hope I don't have to resort to that. Sigh...

Recently I got an e-mail from someone interested in video editors, pointing me to a novel video editor in the works.

The problem: The thing's written in [obscure programming language which is neither Python nor C++]. Oh, you didn't know there was a programming language called [obscure], right? See, THAT's the problem.

I'll quote some text from the other programmer (I'll rephrase all paragraphs to protect the innocent from Google searches):

Basically I'm sick tired of having a non-working video editor and a lot of novel ideas that other editors lack. And I've rewritten it so many times that I've got to get this off my head.All those four years I haven't been programming because of school. But this year might allow me to start working on [editor] again.

Later, I read:

Lately I’ve realized that programming takes too much time from my life.... I almost made it work... I’m not a programmer. Programming in [obscure programming language] is my hobby, not my profession. I don’t get any money from it. It's a problem when programming takes about 80% of my time and I don't have time left for normal activities ... my projects will be put on hiatus. Feel free to ask me for rights to commit to the subversion of [editor]. You can take over the project for now. I hope I’ll get back to programming [editor] with a little more peace of mind after next year [written on December 2007].

And this, ladies and gentlemen, is why the Lone Ranger approach to Open Source programming WILL NEVER WORK. Do you have the slightest idea why so many Sourceforge projects are abandoned? It's like the Open Source Project cemetery. Well here's the reason: Those projects had NO programming teams.

I absolutely refuse to work on a hobby project if I have to do it myself, with tools I have to write myself because there are non-existing tools for an esoteric programming language (let alone a binding to [ famous multimedia library ] ).

Perhaps you'll understand now why I chose to use C++ and wxWidgets. There are at least 10 C++ programmers for every [obscure programming language] programmer. And there are at least 10 Windows users for every Linux user. If I'm going to have a programming team and a live project, I better use things where I can recruit most volunteers.

After all, a Non-Linear Video Editor is not something you can make in your garage.

Thursday, June 19, 2008

Today I received a letter from the MIT guy (the one who never answered my mails) asking me to remove him from the project. However, I have the suspicion that either his request to get in or his request to get out was forged (this means someone got access to his account), because they were in very different tones. Too bad - because we can't allow someone with such a bad security get in the project. Reliability: 50% -> zero.

In other news, I opened a private mailing list for the developers so we can organize ourselves better. I'm also working on the developers guide so anyone can install the software required to compile and run Saya.

Then we'll start working, and hopefully in a couple (ok, maybe 4) weeks we will release version 0.1, code name: Aikuchi. That will mark the end of the planning phase and we will officially become "pre-alpha". Keep in touch! :)

Wednesday, June 18, 2008

An AVClip has a vector of effects.An Effect consists of a map (string, FXParamTimelines). The string denotes the parameter's name.An FXParamTimeline is a map of (integer, string parameter), where the integer represents a point in time.

(Notice that there are no bezier curves in this diagram. But there will be, don't worry about that).

Time is measured in milliseconds, but I'm going to replace the integer with a 64-bit integer so I can represent nanoseconds (this is necessary to be able to move audio clips around with one-sample precision in time).

One thing that I forgot to mention is that none of the data structures I present here, have pointers (pointers are EVIL!). This is so we can duplicate the instances of effects and clips easily. Since the indirection is handled through array (or vector / map) indexes, we won't have segmentation faults due to a badly dereferenced pointer. This also makes serializing and deserializing the data a piece of cake.

When copying a series of clips to the clipboard, I will replace the clip id's with new ones. Simpler than a Jedi mind trick. "These aren't the id#s you're looking for." *Waves hand* No pointer mangling, no headaches.

So this is the reason why I'm trying to focus on the User Interface first. Because it's the task which takes most of the time. And since I already got rid of the wxWidgets-specific classes in this framework, we can choose whatever UI library we want. But I'm sticking with wxWidgets for the user interface.

Who'd guess there was a tool to convert C++ code into UML diagrams? It's called Umbrello.Here's a simplified UML Class diagram for Vidproject and associated classes:

The Video Project has a list of the currently used Resources (video and audio clips - i.e. files ). Additionally, it contains an AVTimeline which contains various sequences. Each sequence contains various audio and video tracks. And each track contains... surprise, a clip. The clip has an index indicating which resource it operates on. And of course, a list of effects and a transition.

Notice that a video clipboard is nothing but a sequence. Convenient, isn't it?

As you can see from this diagram, the memory consumption for these classes is minimal, since they're nothing but sparse data structures.

Now, I don't really know how Cinelerra or other video editors work. But one thing's certain: It's much easier to model the classes after the structure of the data we're going to modify. And this data is a timeline.

I already have implemented saving undo/redos for these classes, and you can impose a memory limit on how many undos/redos you can store in memory. So there goes the high memory requirement. Poof! :)

Tuesday, June 17, 2008

The annoying configuration bugs that had plagued me for the last couple of days were finally solved (note to self: remember to wrap the functions va_start and va_end around vsnprintf the next time - how embarrassing ^^; Oh well... I guess these things happen when you're programming while undersleeping at 4AM in the morning). Another hint, is that whenever you're dealing with char* strings, DO NOT store them in the stack! If anything goes wrong, you won't be able to debug because the stack will be messed up. Use varname = new vartype[num].

AAaaaaaaanyway...

I added a nifty debug log to our wxApp object, so we can finally know what's happening behind the scenes. This is excellent for hard-to-debug cases. And now, back to business, there are tons of things to do:

Fix the updating of recent project menus. I currently use a flag, it's better to use a counter to see whenever it's been updated.

Implement the audio/video presets in the new project menu

Implement project creation and saving with XML.

Keep working on the framework to do some video streaming (Jonathan's beaten me to that already, I must keep up in the race!)

Sunday, June 15, 2008

Turns out there were a lot more wxWidgets functions used in the backend than I expected.

After a few hours of rewriting code, I managed to almost replace all. However, I'm left with a few stubs (I only committed a zipfile with the changes, I don't want to break the build in SVN). Here's a list of the new functions and changes:

ALL wxString references in ProjectManager were replaced by std::string

Since wxWidgets uses the _() macro, all literal strings will have to be surrounded by gettext().

For now it seems that we have to implement the stubs I created. But after that, the code architecture will be much more robust and cleaner, because the ProjectManager and related classes now only depend on the STL and some basic libraries like stdio. This means that if someone wants to make a frontend for the project in QT, it will be much easier for him to do it now.

Saturday, June 14, 2008

I just had realized something. If I want to make Saya UI-Toolkit independent (except the UI frontend), it means I'll have to add another bunch of wrappers. But it's worth it.

So I'll start replacing the wxString references everywhere except in Main.h and Main.cpp for std::string . This will also help me get rid of the _T() and macros. For internationalization, I guess I'll redefine the _() macro to some internal function, used only by the backend.

It also means I'll make an intermediate class to pass the messages (like presenting dialogs). I think I'm going to like this.

Friday, June 13, 2008

We're having technical problems using the SVN server at berlios. It seems I'll have to change the access method from svn+ssh to https. But I'm afraid I can't do it right now, I have this horrible headache. I'll keep you updated.

Update: Fixed. To solve the SVN access Issue, I only had to replace the svn+ssh login method by https. I had to do some directory copying, but it was easy.

Thursday, June 12, 2008

Ah, the joy of multithreading. I really miss the wxMutex and wxMutexLocker classes of wxWidgets, which unfortunately, I can't use for the wrappers (because they're wrappers and therefore they need to be as lightweight as possible. Linking to the wxWidgets library isn't exactly what I want).

The first decision I took was to use the SFML Mutexes class. Unfortunately, I stumbled upon the problem: If I include this for the wrapper, what will happen when one of the plugins actually uses the library? Would I get some duplicate definition? Should I copy the code and rename it?

Forget it. Enter google. I found two great resources for using Mutexes: computing.llnl.gov (which included a copy of the pthreads manpage) and wikipedia. Lucky me, the wikipedia tutorial included a cross-platform version of implementing critical sections! And it's GFDL, alright!

Now let's see if I can find a good example of a cross-platform thread-safe sleep() function.

Update: Done. Now I can go back to work on thread-safety for the audio/video wrappers.

A lot of nice plugins including colour transformations, geometric distortions, basic sound processing and transitions between two clips

A simple interface for developing your own plugins in C++

OpenVIP was designed by a college group as their graduation project. Unfortunately, they couldn't continue their project. That's where I come in. Or should I say we, since there's already a developing team for Saya.

All we're doing is designing a professional user interface around the OpenVIP Video Editor framework. Because the framework is done in C++, designing the frontend is much simpler than starting a video editor from scratch.

The only problem is that OpenVIP is released as GPLv2, while Saya is GPLv3. I've asked the main developer to release the project as GPLv3 so I can add the framework right away. Perhaps his mail got lost, I guess I'll ask again. (Edit: I just got a mail from Antonin Slavik, the OpenVIP project leader. OpenVIP is now GPLv3 or later with linking permission! Road's clear!)

Still... we can't commit the same mistake other developers have by tying themselves to one framework and later realizing it doesn't do what they need. This is why we're making a whole "interface layer" in Saya, so the video processing and playback frameworks can be inserted as plugins. Some things will need to be implemented on our own (or outright stolen from mature frameworks), like threads, mutexes. There's a 90% chance that I'll borrow SFML's system module to deal with this part.

Tuesday, June 10, 2008

* Organizing the team, getting team members. Some members haven't been able to be contacted live - but we got an Adobe Premiere video editing expert / advisor (edit: And tentatively another advisor, who happens to be a Sony Vegas expert and User Interface Nazi ;-) ) , a beta tester, 3 developers and 1 possible developer more (if I manage to get him online). Additionally, an old friend from school who's a college graduate is offering to join the project - but not before he passes a wxWidgets exam I assigned him :)

* Researching on various fields. Nopalin has been able to do his first wxWidgets programs, and is ready to work on the UI. b3rx is in the process of documenting the OpenVIP classes and some classes that have already been designed. I've been researching the use of SFML and have corrected various misunderstandings in the wrapper classes.

* The pluggable multimedia framework has been started. Abstract class AudioOutputDevice has been completed. Abstract class VideoOutputDevice has been started (as a stub). Actual implementations using classes derived from these, remain to be written.

What remains to be done:

- Completing the New Project Dialog and creating new projects.- Creating an A/V streaming class for rendering video and audio on-the-fly.- Creating a plain vanilla video player using the said classes.

I recently found a multimedia library called SFML (Simple and Fast multimedia library). According to the webpage, " Instead of being one big API, SFML rather contains a lot of small packages, that can be chosen and combined according to the intended usage. You can use only the base package to get input and windowing, as well as the full graphics package with sprites and post-effects." Additionally, it's written in C++ and seems to be very flexible.

That sounds just like what we're looking for. However, there is a problem, and it's that the project is (relatively) too young compared to SDL. How stable and reliable it is?

I'm sure that choosing a multimedia library is a problem many multimedia programmers face. They have to base their entire code on choosing one or another library. This is why I decided to design abstract classes as wrappers, so we don't have to face that decision. I hope that with SMFL we can have a working multimedia player soon.

Thursday, June 5, 2008

I just found this awesome post from Michael Dominic explaining his problems with the GStreamer framework.

"GStreamer solves a lot of problems on the GNOME desktop but it doesn’t solve the problem of video editing.

Gst is a playback framework, and for video editing you need editing framework. The later is not, as it’s commonly believed, just a superset of the former. The MLT framework is an interesting example of the “video editing” architecture."

A later comment from a reader explains further:

"I would also join the voices in suggesting that you reconsider your decision with GStreamer. In the last six months GStreamer has become more and more suitable for applications such as Diva, and Gnonlin is a particularly useful component here."

And here I was thinking that I could use GStreamer as a basis for Saya. I think I can explain the concepts with a small diagram:

PiTiVi or some other editorGnonlinGstreamerVideo Hardware

This is, gnonlin is a PLUGIN which goes with GStreamer (if it's not, someone correct me please!). The problem is that this plugin is a video-editing component built upon a low-level playback framework. This is:

Editor UIEditing stuffPlaybackOS / Hardware

and that's a no-no. It's got the logic ALL WRONG!

Instead, what I want is something like this (the higher, the more user interaction) :

In other words, The decoder/files make the MODEL. The Playback/VideoHardware are the PRESENTATION (the GUI is another separate layer of presentation, so they're tied - in a way), and the Editor framework is the CONTROLLER.

Ladies and gentlemen, this is nothing but Model-View-Controller 101.

Any questions?

---------Update (June 23, 2008) : It seems I was mistaken in my assumptions regarding GStreamer and PiTiVi. However, the MVC pattern is still a rule I'm going to follow. Stay tuned.

Wednesday, June 4, 2008

At least I'm not alone. Another guy had more or less the same idea that I had, and around the same time. Clearly this is an indicator that video editing in Linux *DOES* have a problem. His project blog is at http://myvideoeditor.blogspot.com/

Alas, it seems he chose another path (he's going to use GStreamer and Python). Why, oh, why??? I really don't know if he'll succeed, but I asked him to join forces. Sigh, if only we had managed to get in touch earlier. Jonathan claims that C++ is a monster hard to debug. Well, not if you use the latest GDB and Code::Blocks. Programming C++ with Code::Blocks is a breeze ;-)

Luke er Jonathan... JOIN ME, and we shall rule the galaxy together! Well, more or less :P