I’ve spent the last week setting up Rapicorn and Beast with travis-ci.org, a free continuous integration service for Github. Since travis is only available for Github, this means the Beast Git repository (hosted on git.gnome.org) had to be moved (cloned) to Github.

The repos now contain a file .travis.yml that includes the complete build instructions, these need to be kept uptodate if any of the build dependencies change.

By default, travis-ci sets up Ubuntu 12.04 boxes for the continuous builds, but that’s way too old for most dependencies. Luckily there’s a beta program available to use Ubuntu 14.04 ‘trusty’, that can be selected with “dist: trusty”. The g++-4.8 compiler on trusty is still too old to build Beast, so the CI setup currently installs g++-5 from ppa:ubuntu-toolchain-r/test.

As a result, we now have automated test builds running on travis for the Rapicorn and Beast repositories that are triggered on each push command. After each build, the build bot reports success to the #beast IRC channel, and the current status can also be found via the “Build Status” buttons on github: RapicornBeast.

Trying to keep it up, here’s an update on recent developments in Rapicorn and Beast.

Git Branches

For now, Rapicorn and Beast are using Git branches the following way:

Topic branches are created for each change. Where possible, commits should compile and pass all tests (i.e. pass make check installcheck).

Once completed, topic branches are merged into the master branch. For intermediate merges of huge branches, I’ve recently been adding [ongoing] to the merge commit message. As an aside, branch merges should probably be more elaborate in the future to make devlog articles easier to write and potentially more accurate.

Hello and goodbye clang++

Rapicorn C++11 code currently compiles with g++-4.7 and upwards. An initial attempt was made at making the C++11 code compile with clang++-3.4 but the incompatibilities are currently too numerous. A few good fixes have come out of this and are merged into master now, but further work on this branch probably has to wait for a newer clang++ version.

FocusPainter – An ElementPainter that decorates its child according to focus changes.

IDL Improvements

Several changes around Rapicorn’s IDL compiler and support code made it into master recently:

The IDL layer got bind() and connect() mthods (on the ObjectBroker interface). This models the IDL setup phase after the zeromq API. Beast makes use of this when setting up IDL interface layers in the UI and in BSE.

The Python binding was rewritten using Cython. Instead of invoking a heap of generated Python glue code and talking to the message passing interfaces directly, the Python binding now sits on top of the C++ binding. This makes the end results operate much faster, is less complex on the maintenance side and more functional with regards to the Python API offered. As an added bonus, it also eases testing of the C++ bindings.

And just to prove the previous point, the new Cython port uncovered a major issue lurking in the C++ IDL handling of objects in records and sequences. At least since the introduction of remote reference counting, client side object handles and server side object references are implemented and treated in fundamentally different ways. This requires records (struct) and sequences (std::vector) to have separate implementation types on the client and server sides. Thus, the client and server types are now prefixed with ClnT_ and SrvT_ respectively. Newly generated typedef aliases are hiding the prefixes from user code.

The Enum introspection facilities got rewritten so things like the Enum name are also accessible now. This area probably isn’t fully finished yet, for future Any integration a more versatile API is needed still.

Auxillary information for properties is now accessible through an __aida_aux_data__() method on generated interfaces.

Generated records now provide a template method __accept__<>(Visitor) to visit all record fields by value reference and name string. Exemplary visitor implementations are provided to serialize/deserialize records to XML and INI file formats.

BEAST Developments

For the most part, changes in Beast are driving or chasing Rapicorn at the moment. This means that often the tip of Rapicorn master is required to build Beast’s master branch. Here is why:

Beast now uses RAPIDRES(1) to embedd compressed files. Rapicorn::Blob and Rapicorn::Res make these accessible.

Beast now makes use of Rapicorn’s IDL compiler to generate beastrc config structures and to add a new ‘Bse‘ IDL layer into libbse that allows the UI code to interface with Bse objects via C++ interfaces. Of course, lots of additional porting work is needed to complete this.

Beast procedures (a kind of ‘remote method’ implemented in C with lots of boilerplate code) are now migrated to C++ methods one by one which majorly simplifies the code base, but also causes lots of laborious adaptions on the call sites, the UI and the undo system. An excursion into the changes this brings for the undo implementation is provided in DevLog: A day of templates.

The GParamSpec introspection objects for properties that Beast uses for GUI generation can now be constructed from __aida_aux_data__() strings, which enabled the beastrc config structure migration.

An explanatory file HACKING.md was added which describes the ongoing migration efforts and provides help in accessing the object types involved.

What’s next?

For the moment, porting the object system in Beast from GObject to IDL based C++11 interfaces and related procedure, signal and property migrations is keeping me more than busy. I’ll try to focus on completing the majority of work in this area first. But for outlooks, adding a Python REPL might make a good followup step.

]]>https://testbit.eu/rapicorn-idl-moving-into-beast/feed/0DevLog: A day of templateshttps://testbit.eu/a-day-of-templates/
https://testbit.eu/a-day-of-templates/#respondSat, 27 Jun 2015 01:34:58 +0000https://testbit.eu/?p=1308[...]]]>Yesterday I spent some 14+ hours on getting a templated undo method wrapper going.
Just to throw it all away this morning.

Here’s what I was trying to achieve, the C version of BEAST implements undo as follows:

That is, it queues an undo step, that if executed, will call the “insert-part” procedure
on a BseTrack object that inserts a BsePart object at a ‘tick’.
This all happens through a varargs interface with lots of magic behind the scenes. In
particular the reference to ‘part’ is tricky. Future modifications to the BseTrack (or
project) may cause the removal and destruction of the BsePart object involved here.
While the execution of future undo steps will re-create a BsePart to be inserted here
before the step at hand is executed, the ‘part’ object pointer will have to be changed
to the re-created one instead of the destroyed one.
To achieve this, bse_item_push_undo_proc() internally converts the ‘part’ pointer into
a serializable descriptor string that allows to re-identify the BsePart object and the
undo machinery will resolve that before “insert-part” is called.

Now on to C++. I wanted the new pendant in the C++ version of Beast to look like:

In short, I got A, B, C, D, F working after significant efforts.
A is somewhat straight forward with C++11 variable template arguments. C can be accomplished with a C++11 lambda capture list and F involves copying over std::integer_sequence from the C++14 proposals and hacking its std::apply() template to support instance + method calls. Last, D can be implemented in a related fashion to F.
What’s left is B and E, i.e. writing a wrapper that will store and yield ordinary arguments such as int or std::string and convert ItemImpl& derived types back and forth between a string representation.
Probably laborious but doable — or so I thought.

It turns out that because of all the argument and tuple packing hassle (template recursion, integer sequencing and more) involved in implementing A, D, F, it would be hard to pass needed serialization context into Wrapper<>. And what’s much worse is that g++-4.9 started to choke on template errors during the Wrapper<> development, aborting with “confused by earlier errors” after pages and pages of template error messages. clang++-3.4 isn’t yet capable of processing the C++11 used by Rapicorn, so it wasn’t of help here either (I plan on another attempt at porting my C++11 code to be clang++ compatible once I get my hands on a newer clang++ version).
I.e. in the end, I gave up after an overlong day in the middle of E, everything else having been accomplished. g++-4.9 choking was a main let down, but probably even more important is that I had the necessary state and mood to process multiple pages of template error messages yesterday, but the same cannot be expected of every push_undo() user in the future if any push_undo() argument ever mismatches.

This morning, I threw away yesterdays templating excess and within an hour got an alternative interface to work:

That is, this interface is fully type-safe, but the ‘part’ wrapping has to be done manually, which involves writing a small lambda around TrackImpl::insert_part(). If any argument of the lambda or push_undo() calls is erroneous, the compiler will point at a single failing variable assignment in the implementation of push_undo<>() and list the mismatching arguments.
That is much more digestible than multiple template recursion error pages, so it’s a plus on the side of future maintenance.

The short version of push_undo<>() that takes a method pointer instead of a lambda is still available for implementing undo steps that don’t involve object references, incidentally covering the majority of uses.

]]>https://testbit.eu/a-day-of-templates/feed/0Thread-Local-Storage Benchmarkhttps://testbit.eu/thread-local-storage-benchmark/
https://testbit.eu/thread-local-storage-benchmark/#commentsTue, 26 May 2015 16:13:29 +0000http://timj.testbit.eu/?p=928[...]]]>A good while ago at a conference, I got into a debate over the usefulness of TLS (thread-local storage of variables) in performance critical code. Allegedly TLS should be too slow for practical uses, especially for shared libraries.

TLS can be quite useful for context sensitive APIs, here’s a simple example:

For a single threaded program, the above push/pop functions can keep the default background color for widget creation in a static variable. But to allow concurrent widget creation from multiple threads, that variable will have to be managed per-thread, so it needs to become a thread local variable.

Another example is GSlice, a memory allocator that keeps per-thread allocation caches (magazines) for fast successive allocation and deallocation of equally sized memory chunks. While operating within the cache size, only thread local data needs to be accessed to release and reallocate memory chunks. So no other synchronization operations with other threads are needed that could degrade performance.

GCC (I’m using 4.9.1 here), GLibc (2.19), et all have seen a lot of improvements since, so I thought I’d dig out an old benchmark and evaluate how TLS does nowadays. To test the shared library case in particular, I’ve written the benchmark as a patch against Rapicorn and posted it here: thread-local-storage-benchmark.diff.

The following table lists the best results from multiple benchmark runs. The numbers shown are the times for 2 million function calls to fetch a (TLS) pointer of each kind (plus some benchmarking overhead), on a Core-i7 CPU @ 2.80GHz in 64bit mode:

The greatest timing variation in these numbers is within thirty percent (30.6%). In realistic scenarios, the time needed for pointer accesses is influenced by a lot of other more dominant factors, like code locality and data cache faults.

So while it might have been true that TLS had some performance impacts in its infancy, with a modern tool chain on AMD64 Linux, performance is definitely not an issue with the use of thread-local variables.

Here is the count out in nano seconds per pointer access call:

Let me know if there are other platforms that don’t perform as well.

]]>https://testbit.eu/thread-local-storage-benchmark/feed/1DevLog: shared_ptr, resources, eval syntax and morehttps://testbit.eu/shared_ptr-resources-eval-syntax/
https://testbit.eu/shared_ptr-resources-eval-syntax/#respondTue, 05 May 2015 13:16:16 +0000https://testbit.eu/?p=1481[...]]]>Giving in to persistent nagging from Stephen and Stefan about progress updates (thanks guys), I’ll cherry pick some of the branches recently merged into Rapicorn devel for this post. We’ll see if I can keep posting updates more regularly in the future…

Interactive Examples

Following an idea Pippin showed me for his FOSDEM talk, I’ve implemented a very simple small script (merged with the ‘interactive-examples’ branch) to restart an example program if any file of a directory hierarchy changes. This allows “live” demonstrations of widget tree modifications in source code, e.g.:

Shared_ptr widgets

Some while ago, we started to use std::shared_ptr<> to maintain widget reference counts instead of the hand-crafted ref/unref functions that used atomic operations. After several cleanups, we can now also use std::make_shared() to allocate the same memory block for storing the reference count and widget data. Here is an image (originals by Herb Sutter) demonstrating it:

The hand-optimized atomic operations we used previously had some speed advantages, but using shared_ptr was needed to properly implement remote reference counting.

Resources

Since 2003 or so, Beast and later Rapicorn have had the ability to turn any resource file, e.g. PNG icons, into a stream of C char data to be compiled into a program data section for runtime access. The process was rather unordered and adhoc though, i.e. any source file could include char data generated that way, but each case needed its own make rules and support code to access/uncompress and use that data. Lately I did a survey across other projects on how they go about integrating resource files and simplified matters in Rapicorn based on the inspirations I got.
With the merge of the ‘Res’ branch, resource files like icons and XML files have now all been moved under the res/ directory. All files under this subdir are automatically compressed and compiled into the Rapicorn shared library and are accessible through the ‘Res’ resource class. Example:

Blob data = Res ("@res icons/example.png");

Blob objects can be constructed from resources or memory mapped files, they provide size() and data() methods and are automatically memory managed.

New eval syntax

In the recently merged ‘factory-eval-syntax’ branch, we’ve changed the expression evaluation syntax for UI XML files to the following:

<label markup-text="@eval label_variable"></label>

Starting attribute values with ‘@’ has precedence on other platforms and is also useful in other contexts like resources, which allows us to reduce the number of syntax special cases for XML notations.

Additionally, the XML files now support property element syntax, e.g. to set the ‘markup_text’ property of a Label:

This markup is much more natural for complex property values and also has precedence on other platforms.

What’s next

I’m currently knee deep in the guts of new theming code, the majority of which has just started to work but some important bits still need finishing. This also brings some interesting renovation of widget states, which I hope to cover here soon. As always, the Rapicorn Task List contains the most important things to be worked on next. Feedback on missing tasks or opinions on what to prioritize are always appreciated.

]]>https://testbit.eu/shared_ptr-resources-eval-syntax/feed/0Is SSH Insecure?https://testbit.eu/is-ssh-insecure/
https://testbit.eu/is-ssh-insecure/#commentsWed, 31 Dec 2014 03:37:49 +0000https://testbit.eu/?p=1457[...]]]>In the true tradition of previous years, this years 31c3 in Hamburg revealed another bummer about surveillance capacities:

The brief summary is that viable attacks are available to surveillance agencies for PPTP, IPSEC, SSL/TLS and SSH.
New papers reveal that as of 2012, OTR and PGP seem to have resisted decryption attempts.

Several vulnerabilities regarding SSL/TLS have been discovered and fixed in the past years since these papers were created. But at the very least, for state agencies the possibility remains to decrypt individual connections with fake certificates via man-in-the-middle-attacks.

At he conference, I got a chance to discuss this with Jacob after studying some of the Spiegel revelations and since I’ve been asked about this so much I’ll wrap it up here:

The cited papers put an emphasis on breaking other crypto protocols like PPTP and IPSEC. That and even SSL enjoy much more focus than SSH attack possibilities.

Clearly, good attacks are possible against password protected sessions, given lots of computation power or (targeted) password collection databases.

Also 768bit RSA keys are probably nowadays breakable by surveillance agencies and 1024 bit key could be within reach based on revelations about their processing capacities.

Even 2048 bit keys could become approachable given future advances in mathematical attacks or weak random number generators used for key generation as was the case in Debian 2008 (CVE-2008-0166).

Additionally, there always remains the possibility of an undiscovered SSH implementation bug or protocol flaw that’s exploitable for agencies.

Fact is, we don’t yet know enough details about all possible attack surfaces against SSH available to the agencies and we badly need more information to know what infrastructure components remain save and reliable for our day to day work. However we do have an idea about the weak spots that should be avoided.

My personal take away is this:

Never allow password based SSH authentication ever:/etc/ssh/sshd_config: PasswordAuthentication no

Use 4096bit keys for SSH authentication only, I have been doing this for more than 5 years and performance has not been a problem:ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_HOSTNAME -C account@HOSTNAME

In the last months I finally completed and merged a long standing debt into Rapicorn. Ever since the Rapicorn GUI layout & rendering thread got separated from the main application (user) thread, referencing widgets (from the application via the C++ binding or the Python binding) worked mostly due to luck.

I investigated and researched several remote reference counting and distributed garbage collection schemes and many kudos go to Stefan Westerfeld for being able to bounce ideas off of him over time. In the end, the best solution for Rapicorn makes use of several unique features in its remote communication layer:

Only the client (user) thread will ever make two way calls into the server (GUI) thread, i.e. send a function call message and block for a result.

All objects are known to live in the server thread only.

Remote messages/calls are strictly sequenced between the threads, i.e. messages will be delivered and processed sequentially and in order of arrival.

This allows the following scheme:

Any object reference that gets passed from the server (GUI) thread into the client (user) thread enters a server-side reference-table to keep the object alive. I.e. the server thread assumes that clients automatically “ref” new objects that pass the thread boundary.

For any object reference that’s received by a client thread, uses are counted separately on the client-side and once the first object becomes unused, a special message is sent back to the server thread (SEEN_GARBAGE).

At any point after receiving SEEN_GARBAGE, the server thread may opt to initiate the collection of remote object references. The current code has no artificial delays built in and does so immediately (thresholds for delays may be added in the future).

To collect references, the serer thread swaps its reference-table for an empty one and sends out a GARBAGE_SWEEP command.

Upon receiving GARBAGE_SWEEP, the client thread creates a list of all object references it received in the past and for which the client-side use count has dropped to zero. These objects are removed from the client’s internal bookkeeping and the list is sent back to the server as GARBAGE_REPORT.

Upon receiving a GARBAGE_REPORT, corresponding to a previous GARBAGE_SWEEP command, the sever thread has an exact list of references to purge from its previously detached reference-table. Remaining references are merged into currently active table (the one that started empty upon GARBAGE_SWEEP initiation). That way, all object references that have been sent to the client thread but are now unused are discarded, unless they have meanwhile been added into the newly active reference-table.

So far, the scheme works really well. Swapping out the server side reference-tables copes properly with the most tricky case: A (new) object reference traveling from the server to the client (e.g. as part of a get_object() call result), while the client is about to report this very reference as unused in a GARBAGE_REPORT. Such an object reference will be received by the client after its garbage report creation and treated as a genuinely new object reference arriving, similarly to the result of a create_object() call. On the server side it is simply added into the new reference table, so it’ll survive the server receiving the garbage report and subsequent garbage disposal.

The only thing left was figuring out how to automatically test that an object is collected/unreferenced, i.e. write code that checks that objects are gone…
Since we moved to std::shared_ptr for widget reference counting and often use std::make_shared(), there isn’t really any way too generically hook into the last unref of an object to install test code. The best effort test code I came up with can be found in testgc.py. It enables GC layer debug messages, triggers remote object creation + release and then checks the debugging output for corresponding collection messages. Example:

The protocol is fairly efficient in saving bandwitdh and task switches: ref-messages are implicit (never sent), unref-messages are sent only once and support batching (GARBAGE_REPORT). What’s left is the two tiny messages that initiate garbage collection (SEEN_GARBAGE) and synchronize reference counting (GARBAGE_SWEEP). As I hinted earlier, in the future we can introduce arbitrary delays between the two to reduce overhead and increase batching, if that ever becomes necessary.

While its not the most user visible functionality implemented in Rapicorn, it presents an important milestone for reliable toolkit operation and a fundament for future developments like remote calls across process or machine boundaries.

In my previous post Forward Secrecy Encryption for Apache, I’ve described an Apache SSLCipherSuite setup to support forward secrecy which allowed TLS 1.0 and up, avoided SSLv2 but included SSLv3.

With the new PODDLE attack (Padding Oracle On Downgraded Legacy Encryption), SSLv3 (and earlier versions) should generally be avoided. Which means the cipher configurations discussed previously need to be updated.

In a time critical section of a recent project, I came across having to optimize the conversion of three digit US month abbreviations (as commonly found in log files) to integers in C++. That is, for “Jan” yield 1, for “Feb” yield 2, etc, for “Dec” yield 12.

In C++ the simplest implementation probably looks like the following:
std::string string; // input value
std::transform (string.begin(), string.end(), string.begin(), ::toupper);
if (string == "JAN") return 1;
if (string == "FEB") return 2;
// ...
if (string == "DEC") return 12;
return 0; /* mismatch */
In many cases the time required here is fast enough. It is linear in the number of months and depending on the actual value being looked up. But for an optimized inner loop I needed something faster, ideally with running time independent of the actual input value and avoiding branch misses where possible. I could take advantage of a constrained input set, which means the ‘mismatch’ case is never hit in practice.

To summarize:

Find an integer from a fixed set of strings.

Ideal runtime is O(1).

False positives are acceptable, false negatives are not.

That actually sounds a lot like using a hash function. After some bit fiddling, I ended up using a very simple and quick static function that yields a result instantly but may produce false positives. I also had to get rid of using std::string objects to avoid allocation penalties. This is the result:

The hash function operates only on the last 2 ASCII letters of the 3-letter month abbreviations, as these two are sufficient to distinguish between all 12 cases and turn out to yield good hash values. The expression (letter & ~0x20) removes the lowercase ASCII bit, so upper and lower case letters are treated the same without using a potentially costly if-branch. Adding the uppercased ASCII letters modulo 17 yields unique results, so this simple hash value is used to index a 17 element hash table to produce the final result.

In effect, this perfectly detects all 12 month names in 3 letter form and has an almost 30% chance of catching invalid month names, in which case 0 is returned – useful for debugging or assertions if input contracts are broken.

As far as I know, the function is as small as possible given the constraints. Anyone can simplify it further?

The basic need to encrypt digital communication seems to be becoming common sense lately. It probably results from increased public awareness about the number of parties involved in providing the systems required (ISPs, backbone providers, carriers, sysadmins) and the number of parties these days taking an interest in digital communications and activities (advertisers, criminals, state authorities, voyeurs, …). How much to encrypt and to what extend seems to be harder to grasp though.

Default Encryption

A lot of reasons exist why it’s useful to switch to encryption of content and all data transfers by default (Ed). This has been covered elsewhere in more depth, just a very quick summary could be the following:

Not encrypting leads to exposing all online behavior to super easy surveillance by local, foreign and alien authorities, criminals or nosy operators.

Rare use of encryption makes it effectively an alert signal for sensitive messages (e.g. online banking) and an alert signal for interesting personalities (e.g. movement organizers).

Forward Secrecy

Even with good certificates in place and strong encryption algorithms like AES, there’re good reasons to enable use of perfect forward secrecy (PFS) for encrypted connections. One is that PFS means each connection uses a securely generated, separate new encryption key, so that recordings of the connection traffic cannot be decyphered in the future when the server certificate is acquired by an attacker. For the same reason PFS is also useful to mitigate security issues like heartbleed, since the vast majority of generated connection keys are not affected by occasional memory leaks, in contrast to the permanent server certificate that is likely to be exposed by even rare memory leaks.

HTTPS On Subdomains

I investigated the administrative side of things for default encryption of all testbit.eu traffic, which also hosts other sites like rapicorn.org. It turned out a number of measures need to be taken to yield an even mildly acceptable result:

Many browsers out there still only support encrypted HTTPS connections to just one virtual host per IP address (SSL & vhosts).

The affordable (or free) certificates signed by certificate authorities usually just allow one subdomain per certificate (e.g. a free StartSSL certificate covers just www.example.com and example.com).

Site Arrangements

So here’s the list of steps I took to improve privacy on testbit.eu:

I moved everything that was previously hosted on rapicorn.org or beast.testbit.eu into testbit.eu/... subdirectories (leaving a bunch of redirects behind).

Then I obtained a website certificate from startssl.com. I could get my 4096bit certificate signed from them free of charge within a matter of hours. CAcert seems to be the only free alternative, but it’s not supported by major browsers and isn’t even packaged by Debian now.

Having the certificate, I setup my apache to a) reject known-to-be-broken encryption SSL/TLS settings; b) allow a few weak encryption variants to still support old XP browsers; c) give preference to strong encryption schemes with perfect-forward-secrecy.

Last, I setup automatic redirection for all incoming traffic from HTTP to HTTPS on testbit.eu.

Apache PFS Configuration

The Apache configuration bits for TLS with PFS look like the following:

# SSL version 2 has been widely superseded by version 3 or TLS
SSLProtocol All -SSLv2
# Compression is rarely supported and vulnerable, see CRIME attack
SSLCompression Off
# Preferred Cipher suite selection favoring PFS
SSLCipherSuite EECDH+AESGCM:EDH+AESGCM:HIGH:!MEDIUM:!LOW:!SSLv2:!aNULL:!PSK
# Enable picking ciphers in the above order
SSLHonorCipherOrder on

Even stricter variants would be disabling 3DES and SSLv3. But it turns out that IE8 on XP still needs 3DES. Also Firefox-25 on Ubuntu-1304 still needs SSLv3. I still mean to support both for a few more months. This is the stricter configuration:

During recent weeks, I’ve started to create a new tool “Tobin” to generate website statistics for a number of sites I’m administrating or helping with. I’ve used programs like Webalizer, Visitors, Google Analytics and others for a long time, but there’re some correlations and relationships hidden in web server log files that are hard or close to impossible to visualize with these tools.

The Javascript injections required by Google Analytics are not always acceptable and fall short of accounting for all traffic and transfer volume. Also some of the log files I’m dealing with are orders of magnitudes larger than available memory, so the processing and aggregation algorithms used need to be memory efficient.

Here is what Tobin currently does:

Input records are read and sorted on disk, inputs are filtered for a specific year.

A 30 minute window is used to determine visits via unique IP-address and UserAgent.

Statistics such as per hour accounting and geographical origin are collected.

Top-50 charts and graphs are generated in an HTML report from the collected statistics.

There is lots of room for future improvements, e.g. creation of additional modules for new charts and graphs, possibly accounting across multiple years, use of intermediate files to speed up processing and more. In any case, the current state works well for giant log files and already provides interesting graphs. The code right now is alpha quality, i.e. ready for a technical preview but it might still have some quirks. Feedback is welcome.

Next Friday I’ll be giving a talk on Open Source In Business at the Campus Party Europe conference in the O2 arena, London. The talk is part of the Free Software Track at 14:00 on the Archimedes stage. I’m there the entire week and will be happy to meet up, so feel free to drop me a line in case you happen to be around the venue as well.

The picture above shows the London Southwark Bridge viewed from Millenium Footbridge with the London Tower Bridge in the background. Last week’s weather has been exceptional for sightseeing in London, so there were loads of great chances to visit that vast number of historical places and monuments present in London. My G+ stream has several examples of the pictures taken. I honestly hope the good weather continues and allows everyone to enjoy a great conference!

Would you want to invest hours or days into automake logic without a use case?

For two of the last software releases I did, I was facing this question. Let me give a bit of background. Recently the documentation generation for Beast and Rapicorn fully switched over to Doxygen. This has brought a number of advantages, such as graphs for the C++ inheritance of classes, build tests of documentation example code and integration with the Python documentation. What was left for improvement was a simplification of the build process and logic involved however.

Generating polished documentation is time consuming

Maintaining the documentation builds has become increasingly complex. One of the things adding to the complexity are increased dependencies, external tools are required for a number of features: E.g. Doxygen (and thus Qt) is required to build the main docs, dot is required for all sorts of graph generation, python scripts are used to auto-extract some documentation bits, rsync is used for incremental updates of the documentation, Git is used to extract interesting meta information like the documentation build version number or the commit logs, etc.

More complexity for tarballs?

For the next release, I faced the task of looking into making the documentation generation rules work for tarballs, outside of the Git repository. That means building in an environment significantly different from the usual development setup and toolchain (of which Git has become an important part). At the very least, this was required:

Creating autoconf rules to check for Doxygen and all its dependencies.

Require users to have a working Qt installation in order to build Rapicorn.

Deriving a documentation build version id without access to Git.

Getting the build dependencies right so we auto-build when Git is around but don’t break when Git’s not around.

All of this just for the little gain of enabling the normal documentation re-generation for someone wanting to start development off a tarball release.
Development based on tarballs? Is this a modern use case?

Development happens in Git

During this year’s LinuxTag, I’ve taken the chance to enter discussions and get feedback on development habits in 2013. Development based on tarballs certainly was the norm when I started in Free Software & Open Source, that was 1996. It’s totally not the case these days. A large number of projects moved to Git or the likes. Rapicorn and Beast have been moved to Git several years ago, we adopted the commit style of the Linux kernel and a GNU-style ChangeLog plus commit hash ids is auto-generated from Git for the tarballs.
Utilizing the meta information for a project living in Git comes naturally as time passes and projects get more familiar with Git. Examples are signed tags, scripts around branch-/merge-conventions, history greping or symbolic version id generation. Git also significantly improves spin-off developments which is why development of Git hosted projects generally happens in Git branches or Git clones these days. Sites like github encourage forking and pulling, going back to the inconveniences of tarball based development baring any history would be a giant leap backwards. In fact, these days tarballs serve as little more than a transport container for a specific snapshot of a Git repository.

Shipping pre-built Documentation

Taking a step back, it’d seem easier to avoid the hassle of adapting all the documentation build logic to work both ways, with and without Git, by simply including a copy of the readily built result into the tarball. Like everything, there’s a downside here as well of course, tarball size will increase significantly. Just how significantly? The actual size can make or break the deal, e.g. if it changed by orders of magnitude. Let’s take a look:

6.0M – beast-0.8.0.tar.bz2

23.0M – beast-full-docs.tar-bz2

Uhhh, that’s a lot. All the documentation for the next release totals around almost four times that of the last tarball size. That’s a bit excessive, can we do better?

It turns out that a large portion of the space in a full Doxygen HTML build is actually used up by images. Not the biggest chunk but a large one nevertheless, for the above example, we’re looking at:

23M – du -hc full-docs/*png

73M – du -hc full-docs/

So, 23 out of 73 MB for the images, that’s 32%. Doxygen doesn’t make it too hard to build without images, it just needs two configuration settings HAVE_DOT = NO and CLASS_DIAGRAMS = NO. Rebuilding the docs without any images also removes a number of image references, so we end up with:

42M – slim-docs/

That’s a 42% reduction in documentation size. Actually that’s just plain text documentation now, without any pre-compressed PNG images. That means bzip2 could do a decent job at it, let’s give it a try:

2.4M – beast-slim-docs.tar-bz2

Wow, that went better than expected, we’re just talking about 40% of the source code tarball at this point. Definitely acceptable, here’re the numbers for the release candidates in direct comparison, with and without pre-built documentation:

6.1M – beast-no-docs-0.8.1-rc1.tar.bz2

8.6M – beast-full-docs-0.8.1-rc1.tar.bz2

Disable Documentation generation rules in tarballs

Now that we’ve established that shipping documentation without graphs results in an acceptable tarball size increase, it’s easy to make the call to include full documentations with tarball releases. As a nice side effect, auto-generation of the documentation in tarballs can be disabled (not having the Git tree and other tools available, it’d be prone to fail anway). The only thing to watch out for is a srcdir!=builddir case with automake, as in Git trees documentation is build inside builddir, while it’s shipped and available from within srcdir in tarballs.

Pros and Cons for shipping documentation

Con: Tarball source changes cannot be reflected in docs. This mostly affects packagers, it’d be nice to receive substantial patches in upstream as a remedy.

Pro: The build logic is significantly simplified, allowing a hard dependency on Git and skipping complex conditionals for tool availability.

Pro: Build time and complexity from tarballs is reduced. A nice side effect, considering the variety of documentation tools out there, sgml-tools, doxygen, gtk-doc, etc.

For me the pros in this case clearly outweigh the cons. I’m happy to hear about pros and cons I might have missed.

Prior Art?

Looking around the web for cases of other projects doing this didn’t turn up too many examples. There’s some probability that most projects don’t yet trade documentation generation rules for pre-generated documentation in tarballs.

If you know projects that turned to pre-generated documentation, please let me know about them.

I’m also very interested in end-user and packager’s opinions on this. Also, do people know about other materials that projects include pre-built in tarballs? And without the means to regenerate everything from just the tarballs?

During a conference some while ago, Jacob Appelbaum gave a talk on the usefulness of the Tor project, allowing you to browse anonymously, liberating speech online, enabling web access in censored countries, etc.

Jacob described how the anonymizing Tor network consists of many machines world wide that use encryption and run the Tor software, which are routing internet traffic and on the way anonymize it, and then traffic leaves the network at some random host so the original sender cannot be traced back. These hosts are called “exit nodes”.

At the end of his talk, he prompted the audience:Why don’t you run an exit node yet?
I had been using Tor in the past on and off, and while I couldn’t agree more with the privacy goals and anti-censorship measures outlined, I never setup an exit node to help the network. And I do admin quite a number of hosted machines that have idle bandwidth available…

It took me a while to get round to it, but some months after that I started to set up the first exit node on a hosted virtual server. It took a while to get it all going, I made sure I read up the legal implications of running it in Germany, setup disclaimers on the host for people checking it’s port 80, etc. After half a day or so, I had it going, watched in the logs how it connected to the network and… let it run.

Traffic came in slowly at first, but after 1 or 2 days, the node’s presence had propagated through the net and it started to max out CPU and bandwidth limits as configured. So far so good, I was happy helping people all over the world browsing the net anonymously and especially helping folks in countries with internet censorship to access all the net. Great!

Or so I thought at least. It only took some 5 or so days for me to get an official notice to cease network activity on this host immediately. Complaints about Copyright infringement were cited as the reason. Turned out that the majority of the “liberating” traffic I was relaying were torrenting copyrighted material. I had checked out the Tor guidelines in advance, which are correctly outlining that in Germany the TMG (law on telecommunication media) paragraphs §8 and §15 are actually protecting me as a traffic router from liability for the actual traffic contents, so initially I assumed I’d be fine in case of claims.

It turned out the notice had a twist to it. It was actually my virtual server provider who sent that notice on behalf of a complaining party and argued that I was in violation of their general terms and conditions for purchasing hosting services. Checking those, the conditions read:Use of the server to provide anonymity services is excluded.
Regardless of the TMG, I was in violation of the hosting provider’s terms and conditions which allowed premature termination of the hosting contract. At that point I had no choice but stopping the Tor services on this hosting instance.

All in all a dissatisfying experience, but at least I could answer Jacob’s question now:I’m not running an exit node because it’s not uncommon for German providers to exclude the use of anonymity services on the merits.
I actually got back to Jacob in Email and suggested that a note be added to the TorExitGuidelines wiki page so future contributors know to check out the terms and conditions of their hosting services. It seems my request has been ignored up to this day, for one reason or another.

I’d still like to support the Tor network however, so for all savvy readers out there, I’m asking:

Do you have any provider recommendations where running Tor exit nodes is not an issue? (In Germany perhaps?)

Is it at all feasible to be running Tor exit nodes in Germany without having to set a legal budget aside to defend yourself against claims?

In the last few days I finished reading the “Black Swan” by Nassim Nicholas Taleb. Around last January I saw Günther Palfinger mentioning it in my G+ stream, looked it up and bought it.

At first, the book seemed to present some interesting ideas on error statistics and the first 20 or 30 pages are giving good examples for conscious knowledge we posses but don’t apply in every day actions. Not having a trading history like the author, I found reading further until around page 100 to be a bit of a drag. Luckily I kept on, because after that Taleb started to finally get interesting for me.

Once upon a time…
One of the lectures I attended at university touched on black box analysis (in the context of modelling and implementation for computer programs). At first of course the usual and expected or known input/output behavior is noted, e.g. calculus it may perform or pattern recognition or any other domain specific function. But in order to find out hints about how it’s implemented, short of inspecting the guts which a black box won’t allow for, one needs to look at error behavior. I.e. examine the outputs in response to invalid/undefined/distorted/erroneous/unusual inputs and assorted response times. For a simple example, read a sheet of text and start rotating it while you continue reading. For untrained people, reading speed slows down as the rotation angle increases, indicating that the brain engages in counter rotation transformations which are linear in complexity with increasing angles.

At that point I started to develop an interest in error analysis and research around that field, e.g. leading to discoveries like the research around “error-friendliness” in technological or biological systems or discoveries of studies on human behavior which implies corollaries like:

Displaying heuristic behavior, people must make errors by design. So trying to eliminate or punish all human error is futile, aiming for robustness and learning from errors instead is much better.

Perfectionism is anti-evolutionary, it is a dead end not worth striving for. For something “perfect” lacks flexibility, creativity, robustness and cannot be improved upon.

A Black Swan?
Now “Black Swan” defines the notion of a high-impact, low-probability event, e.g. occurring in financial trading, people’s wealth or popularity – events from an extreme realm. That’s in contrast to normally distributed encounters like outcomes of a dice game, people’s body size or the number of someone’s relatives – encounters from a mediocre realm.

From Mediocre…
Here’s a short explanation for the mediocre realms. Rolling a regular dice will never give a number higher than 6 no matter how often it’s thrown. In fact, the more it’s thrown, the more even it’s numbers are distributed and the clearer its average emerges. Measuring people’s weight or number of relatives shows a similar pattern to throwing a dice, the more measurements are encountered the more certain the average becomes. Any new encounter is going to have lesser and lesser impact on the average of the total as the number of measurements increases.

To Extreme…
On the other hand there are the extreme realms. In trading or wealth or popularity, a single encounter can outweigh the rest of the distribution by several orders of magnitude. Most people have an annual income of less than $100k, but the tiny fraction of society that earns more in annual income possesses more than 50% of the entire distribution of wealth. A similar pattern exists with popularity, only very few people are so popular that they’re known by hundreds of thousands or maybe millions of people. But only very very few people are super popular so they’re known by billions. Averaging over a given set only works for so long, until a high-impact “outlier” is encountered that dominates the entire distribution. Averaging the popularity of hundreds of thousands of farmers, industrial workers or local mayors cannot account for the impact on the total popularity distribution by the encounter of a single Mahatma Gandhi.

On ErrorsTaleb is spending a lot of time in the book on condemning the application of the Gauss distribution in fields that are prone to extreme encounters especially economics. Rightfully so, but I would have enjoyed learning more about examples of fields that are from the extreme realms and not widely recognized as such. The crux of the inapplicability of the Gauss distribution in the extreme realms lies in two things:

Small probabilities are not accurately computable from sample data, at least not accurately enough to allow for precise decision making. The reason is simple, since the probabilities of rare events are very small, there simply cannot be enough data present to match any distribution model with high confidence.

Rare events that have huge impact, enough impact to outweigh the cumulative effect of all other distribution data, are fundamentally non-Gaussian. Fractal distributions may be useful to retrofit a model to such data, but don’t allow for accurate predictability. We simply need to integrate the randomness and uncertainty of these events into our decision making process.

Aggravation in the Modern Age
Now Taleb very forcefully articulates what he thinks about economists applying mathematical tools from the mediocre realms (Gauss distribution, averaging, disguising uncertain forecasts as “risk measurements”, etc) to extreme realm encounters like trade results and if you look for that, you’ll find plenty of well pointed criticism in that book. But what struck me as very interesting and a new excavation in an analytical sense is that our trends towards globalisation and high interconnectedness which yield ever growing and increasingly bigger entities (bigger corporations, bigger banks, quicker and greater popularity, etc) are building up the potential for rare events to have higher and higher impacts. E.g. an eccentric pop song can make you much more popular these days on the Internet than TV could do for you 20 years ago. A small number of highly interconnected banks these days have become so big that they “cannot be allowed to fail”.

We are all Human
Considering how humans are essentially functioning as heuristic and not precise systems (and for good reasons), every human inevitably will commit mistakes and errors at some point and to some lesser or larger degree. Now admitting we all error once in a while, exercising a small miscalculation during grocery shopping, buying a family house, budgeting a 100 people company, leading a multi-million people country or operating a multi-trillion currency reserve bank has of course vastly different consequences.

What really got me
So the increasing centralisation and increasing growth of giant entities ensures that todays and future miscalculations are disproportionally exponentiated. In addition, use of the wrong mathematical tools ensures miscalculations won’t be small, won’t be rare, their frequency is likely to increase.

Notably, global connectedness alerts the conditions for Black Swan creation, both in increasing frequency and increasing impact whether positive or negative. That’s like our modern society is trying to balance a growing upside down pyramid of large, ever increasing items on top of its head. At some point it must collapse and that’s going to hurt, a lot!

Take Away
The third edition of the book closes with essays and commentary that Taleb wrote after the the first edition and in response to critics and curios questions. I’m always looking for relating things to practical applications, so I’m glad I got the third edition and can provide my personal highlights to take away from Taleb’s insights:

Avoid predicting rare eventsThe frequency of rare events cannot be estimated from empirical observation because of their very rareness (i.e. calculation error margin becomes too big). Thus the probability of high impact rare events cannot be computed with certainty, but because of the high impact it’s not affordable to ignore them.

Focus on impact but not probability
It’s not useful to focus on the probability of rare events since that’s uncertain. It’s useful to focus on the potential impact instead. That can mean to identify hidden risks or to invest small efforts to enable potentially big gains. I.e. always consider the return-on-investment ratio of activities.

Rare events are not alike (atypical)
Since probability and accurate impact of remote events are not computable, reliance on rare impacts of specific size or around specific times is doomed to fail you. Consequently, beware of others making related predictions and/or others relying them.

Strive for variety in your endeavorsAvoiding overspecialization, learning to love redundancy as well as broadening one’s stakes reduces the effect any single “bad” Black Swan event can have (increases robustness) and variety might enable some positive Black Swan events as well.

What’s next?
The Black Swan idea sets the stage for further investigations, especially investigation of new fields for applicability of the idea. Fortunately, Nassim Taleb continues his research work and has meanwhile published a new book “Antifragile – Things that Gain from Disorder”. It’s already lying next to me while I’m typing and I’m happily looking forward to reading it.

The notion of incomputable rare but consequential events or “errors” is so ubiquitous that many other fields should benefit from applying “Black Swan”- or Antifragile-classifications and corresponding insights. Nassim’s idea to increase decentralization on the state level to combat escalation of error potentials at centralized institutions has very concrete applications at the software project management level as well. In fact the Open Source Software community has long benefited from decentralized development models and through natural organization avoided giant pitfall creation that occur with top-down waterfall development processes.

Algorithms may be another field where the classifications could be very useful. Most computer algorithm implementations are fragile due to high optimization for efficiency. Identifying these can help in making implementations more robust, e.g. by adding checks for inputs and defining sensible fallback behavior in error scenarios. Identifying and developing new algorithms with antifragility in mind should be most interesting however, good examples are all sorts of caches (they adapt according to request rates and serve cached bits faster), or training of pattern recognition components where the usefulness rises and falls with the variety and size of the input data sets.

Conclusion
The book “Black Swan” is definitely a highly recommended read. However make sure you get the third edition that has lots of very valuable treatment added on at the end, and don’t hesitate to skip a chapter or two if you find the text too involved or side tracking every once in a while. Taleb himself gives advice in several places in the third edition about sections readers might want to skip over.

Have you read the “Black Swan” also or heard of it? I’d love to hear if you’ve learned from this or think it’s all nonsense. And make sure to let me know if you’ve encountered Black Swans in contexts that Nassim Taleb has not covered!

About

Determine candidates and delete from a set of directories containing aging backups. As a follow up to the release of sayebackup.sh last December, here’s a complimentary tool we’re using at Lanedo. Suppose a number of backup directories have piled up after a while, using sayebackup.sh or any other tool that creates time stamped file names:

Which file should be deleted once the backup device starts to fill up? Sayepurge parses the timestamps from the names of this set of backup directories, computes the time deltas, and determines good deletion candidates so that backups are spaced out over time most evenly. The exact behavior can be tuned by specifying the number of recent files to guard against deletion (-g), the number of historic backups to keep around (-k) and the maximum number of deletions for any given run (-d). In the above set of files, the two backups from 2011-07-07 are only 6h apart, so they make good purging candidates, example:

For day to day use, it makes sense to use both tools combined e.g. via crontab. Here’s a sample command to perform daily backups of /etc/ and then keep 6 directories worth of daily backups stored in a toplevel directory for backups:

Resources

Usage

Usage: sayepurge.sh [options] sources...
OPTIONS:
--inc merge incremental backups
-g <nguarded> recent files to guard (8)
-k <nkeeps> non-recent to keep (8)
-d <maxdelet> maximum number of deletions
-C <dir> backup directory
-o <prefix> output directory name (default: 'bak')
-q, --quiet suppress progress information
--fake only simulate deletions or merges
-L list all backup files with delta times
DESCRIPTION:
Delete candidates from a set of aging backups to spread backups most evenly over time, based on time stamps embedded in directory names. Backups older than <nguarded> are purged, so that only <nkeeps> backups remain. In other words, the number of backups is reduced to <nguarded> + <nkeeps>, where <nguarded> are the most recent backups. The puring logic will always pick the backup with the shortest time distance to other backups. Thus, the number of <nkeeps> remaining backups is most evenly distributed across the total time period within which backups have been created. Purging of incremental backups happens via merging of newly created files into the backups predecessor. Thus merged incrementals may contain newly created files from after the incremental backups creation time, but the function of reverse incremental backups is fully preserved. Merged incrementals use a different file name ending (-xinc).

Performance of a C++11 Signal System

First, a quick intro for for the uninitiated, signals in this context are structures that maintain a lists of callback functions with arbitrary arguments and assorted reentrant machinery to modify the callback lists and calling the callbacks. These allow customization of object behavior in response to signal emissions by the object (i.e. notifying the callbacks by means of invocations).

Over the years, I have rewritten each of GtkSignal, GSignal and Rapicorn::Signal at least once, but most of that is long a time ago, some more than a decade. With the advent of lambdas, template argument lists and std::function in C++11, it became time for me to dive into rewriting a signal system once again.

So for the task at hand, which is mainly to update the Rapicorn signal system to something that fits in nicely with C++11, I’ve settled on the most common signal system requirements:

Signals need to support arbitrary argument lists.

Signals need to provide single-threaded reentrancy, i.e. it must be possible to connect and disconnect signal handlers and re-emit a signal while it is being emitted in the same thread. This one is absolutely crucial for any kind of callback list invocation that’s meant to be remotely reliable.

Signals should support non-void return values (of little importance in Rapicorn but widely used elsewhere).

Signals can have return values, so they should support collectors (i.e. GSignal accumulators or boost::signal combiners) that control which handlers are called and what is returned from the emission.

Signals should have only moderate memory impact on class instances, because at runtime many instances that support signal emissions will actually have 0 handlers connected.

For me, the result is pretty impressive. With C++11 a simple signal system that fullfils all of the above requirements can be implemented in less than 300 lines in a few hours, without the need to resort to any preprocessor magic, scripted code generation or libffi.

I say “simple”, because over the years I’ve come to realize that many of the bells and whistles as implemented in GSignal or boost::signal2 don’t matter much in my practical day to day programming, such as the abilities to block specific signal handlers, automated tracking of signal handler argument lifetimes, emissions details, restarts, cancellations, cross-thread emissions, etc.

Beyond the simplicity that C++11 allows, it’s of course the performance that is most interesting. The old Rapicorn signal system (C++03) comes with its own set of callback wrappers named “slot” which support between 0 and 16 arguments, this is essentially mimicking std::function. The new C++11 std::function implementation in contrast is opaque to me, and supports an unlimited number of arguments, so I was especially curious to see the performance of a signal system based on it.

I wrote a simple benchmark that just measures the times for a large number of signal emissions with negligible time spent in the actual handler.

I.e. the signal handler just does a simple uint64_t addition and returns. While the scope of this benchmark is clearly very limited, it serves quite well to give an impression of the overhead associated with the emission of a signal system, which is the most common performance relevant aspect in practical use.

Without further ado, here are the results of the time spent per emission (less is better) and memory overhead for an unconnected signal (less is better):

Signal System

Emit() in nanoseconds

Static Overhead

Dynamic Overhead

GLib GSignal

341.156931ns

0

0

Rapicorn::Signal, old

178.595930ns

64

0

boost::signal2

92.143549ns

24

400 (=265+7+8*16)

boost::signal

62.679386ns

40

392 (=296+6*16)

Simple::Signal, C++11

8.599794ns

8

0

Plain Callback

1.878826ns

–

–

Here, “Plain Callback” indicates the time spent on the actual workload, i.e. without any signal system overhead, all measured on an Intel Core i7 at 2.8GHz. Considering the workload, the performance of the C++11 Signals is probably close to ideal, I’m more than happy with its performance. I’m also severely impressed with the speed that std::function allows for, I was originally expecting it to be at least a magnitude larger.

The memory overhead gives accounts on a 64bit platform for a signal with 0 connections after its constructor has been called. The “static overhead” is what’s usually embedded in a C++ instance, the “dynamic overhead” is what the embedded signal allocates with operator new in its constructor (the size calculations correspond to effective heap usage, including malloc boundary marks).

The reason GLib’s GSignal has 0 static and 0 dynamic overhead is that it keeps track of signals and handlers in a hash table and sorted arrays, which only consume memory per (instance, signal, handler) triplet, i.e. instances without any signal handlers really have 0 overall memory impact.

Summary:

If you need inbuilt thread safety plus other bells and can spare lots of memory per signal, boost::signal2 is the best choice.

For tight scenarios without any spare byte per instance, GSignal will treat your memory best.

If you just need raw emission speed and can spare the extra whistles, the C++11 single-file simplesignal.cc excels.

For the interested, the brief C++11 signal system implementation can be found here: simplesignal.ccThe API docs for the version that went into Rapicorn are available here: aidasignal.hh

PS: In retrospect I need to add, this day and age, the better trade-off for Glib could be one or two pointers consumed per instance and signal, if those allowed emission optimizations by a factor of 3 to 5. However, given its complexity and number of wrapping layers involved, this might be hard to accomplish.

About

Due to popular request, I’m putting up a polished version of the backup script that we’ve been using over the years at Lanedo to backup our systems remotely. This script uses a special feature of rsync(1) v2.6.4 for the creation of backups which share storage space with previous backups by hard-linking files.The various options needed for rsync and ssh to minimize transfer bandwidth over the Internet, time-stamping for the backups and handling of several rsync oddities warranted encapsulation of the logic into a dedicated script.

Usage

Usage: sayebackup.sh [options] sources...OPTIONS: --inc make reverse incremental backup --dry run and show rsync with --dry-run option --help print usage summary -C <dir> backup directory (default: '.') -E <exclfile> file with rsync exclude list -l <account> ssh user name to use (see ssh(1) -l) -i <identity> ssh identity key file to use (see ssh(1) -i) -P <sshport> ssh port to use on the remote system -L <linkdest> hardlink dest files from <linkdest>/ -o <prefix> output directory name (default: 'bak') -q, --quiet suppress progress information -c perform checksum based file content comparisons --one-file-system -x disable crossing of filesystem boundaries --version script and rsync versionsDESCRIPTION: This script creates full or reverse incremental backups using the rsync(1) command. Backup directory names contain the date and time of each backup run to allow sorting and selective pruning. At the end of each successful backup run, a symlink '*-current' is updated to always point at the latest backup. To reduce remote file transfers, the '-L' option can be used (possibly multiple times) to specify existing local file trees from which files will be hard-linked into the backup.Full Backups: Upon each invocation, a new backup directory is created that contains all files of the source system. Hard links are created to files of previous backups where possible, so extra storage space is only required for contents that changed between backups.Incremental Backups: In incremental mode, the most recent backup is always a full backup, while the previous full backup is degraded to a reverse incremental backup, which only contains differences between the current and the last backup.RSYNC_BINARY Environment variable used to override the rsync binary path.

See Also

]]>https://testbit.eu/sayebackup-sh-deduplicating-backups-with-rsync/feed/6ListItemFilter Mediawiki Extensionhttps://testbit.eu/listitemfilter-mediawiki-extension/
https://testbit.eu/listitemfilter-mediawiki-extension/#commentsFri, 23 Nov 2012 17:58:17 +0000http://timj.testbit.eu/?p=846For a while now, I’ve been maintaining my todo lists as backlogs in a Mediawiki repository. I’m regularly deriving sprints from these backlogs for my current task lists. This means identifying important or urgent items that can be addressed next, for really huge backlogs this can be quite tedious.

A SpecialPage extension that I’ve recently implemented now helps me through the process. Using it, I’m automatically getting a filtered list of all “IMPORTANT:”, “URGENT:” or otherwise classified list items. The special page can be used per-se or via template inclusion from another wiki page. The extension page at mediawiki.org has more details.

Like every year, I am driving to Berlin this week to attend LinuxTag 2012 to attend the excellent program. If you want to meet up and chat about projects, technologies, Free Software or other things, send me an email or leave a comment with this post and we will arrange for it.

About Testbit Tools The ‘Testbit Tools’ package contains tools proven to be useful during the development of several Testbit and Lanedo projects. The tools are Free Software and can be redistributed under the GNU GPLv3+.

This release features the addition of buglist.py, useful to aid in report and summary generation from your favorite bugzilla.

What’s this? Wikihtml2man is an easy to use converter that parses HTML sources, normally originating from a Mediawiki page, and generates Unix Manual Page sources based on it (also referred to as html2man or wiki2man converter). It allows developing project documentation online, e.g. by collaborating in a wiki. It is released as free software under the GNU GPLv3. Technical details are given in its manual page: Wikihtml2man.1.

Why move documentation online? Google turns up a few alternative implementations, but none seem to be designed as a general purpose tool. With the ubiquituous presence of wikis on the web these days and the ease of content authoring they provide, we’ve decided to move manual page authoring online for the Beast project. Using Mediawiki, manual pages turn out to be very easily created in a wiki, all that’s then needed is a backend tool that can generate Manual Page sources from a wiki page. Wikihtml2man provides this functionality based on the HTML generated from wiki pages, it can convert a prerendered HTML file or download the wiki page from a specific URL. HTML has been choosen as input format to support arbitrary wiki features like page inclusion or macro expansion and to potentially allow page generation from other wikis than MediaWiki. Since wikihtml2man is based purely on HTML input, it is of course also possible to write the Manual Page in raw HTML, using tags such as h1, strong, dt, dd, li, etc, but that’s really much less convenient to use than a regular wiki engine.

What are the benefits? For Beast, the benefits of moving some project documentation into an online wiki are:

We increase editability by lifting review requirements.

We are getting quicker edit/view turnarounds, e.g. through use of page preview functionality in wikis.

We allow assimilation of user contributions from non-programmers for our documentation.

Easier editability may lead to richer documentation and possibly better/frequently updated documentation.

What are the downsides? We have only recently moved our pages online and still need to gather some experience with the process. So far possible downsides we see are:

Sources and documentation can more easily get out of sync if they don’t reside in the same tree. We hope to be mitigating this by increasing documentation update frequencies.

Confusion about revision synchronization, with the source code using a different versioning system than the online wiki. We are currently pondering automated re-integration into the tree to counteract this problem.

How to use it? Here’s wikihtml2man in action, converting its own manual page and rendering it through man(1):

Have feedback or questions? If you can put wikihtml2man to good use, have problems running it or other ideas about it, feel free to drop me a line about it. Alternatively you can also add your feedback and any feature requests to the Feature Requests page (a forum will be created if there’s any actual demand).

What’s related? We would also like to hear from other people involved in projects that are using/considering wikis to build production documentation online (e.g. in manners similar to Wikipedia). So please leave a comment and tell us about it if you do something similar.

Like every year, I am driving to Berlin this week to attend LinuxTag 2011 to attend the excellent program. If you want to meet up and chat about projects, technologies, Free Software or other things, send me an email or leave a comment with this post and we will arrange for it.

People are often telling me they are good at multitasking, i.e. handling multiple things at once and performing well at doing so. Now, the human brain can only make a single conscious decision at a time. To understand this, we need to consider that making a conscious decision requires attention, and the very concept of attention means activating relevant information contexts for an observation or decision making and inhibiting other irrelevant information.

The suppression involved in attention control makes it harder for us to continue with a previously executed task, this is why interruptions affect our work flows badly, such as an incoming call, SMS or a door bell. Even just making a decision on whether to take a call already requires attention diversion.

Related, processing emails or surfing while talking to someone on the phone results in bad performance on both tasks, because the attention required for each, necessarily suppresses resources needed by the second task. Now some actions don’t suffer from this competition, we can walk and breathe or balance ourselves fine while paying full attention to a conversation. That’s because we have learned early on in our lives to automate these seemingly mundane tasks, so they don’t require our conscious attention at this point.

Studies [1][2] have shown time and again, that working on a single task in isolation yields vastly better results and in a shorter time frame when frequent context switches are avoided. This can be further optimized by training in concentration techniques, such as breath meditation, autogenic training or muscle relaxation.

Here’s a number of tips that will help to put these findings to practical use:

Let go of the idea of permanent reachability, nothing is so urgent that it cannot wait the extra hour to be handled efficiently.

Make up your own mind about when to process emails, SMS, IM, news, voice messages.

Start growing a habit of processing things in batches, e.g. walk through a list of needed phone calls in succession, compose related replies in batches, first queue and later process multiple pending reviews at once, queue research tasks and walk through them in a separate browsing session, etc.

Enforce non-availability periods where you cannot be interrupted and may concentrate on tasks of your choice for an extended period.

Everybody is invited to come and visit us in hall 2, booth D44/124, in the open source park. We will give introductions to our services, talk about current and future developments around GTK+ and Tracker and anything you want to approach us with.

MediaWiki is a pretty fast piece of software out of the box. It’s written in PHP and covers a lot of features, so it can’t serve pages in 0 time, but it’s reasonably well written and allows use of PHP accelerators or caches in most cases. Since it’s primarily developed for Wikipedia, it’s optimized for high performance deployments, caching support is available for Squid, Varnish and plain files.

For small scale use cases like private or intranet hosts, running MediaWiki uncached will work fine. But once it’s exposed to the Internet, regularly crawled and might receive links from other popular sites, serving only a handful of pages per second is quickly not enough. A very simple but effective measure to take in this scenario is the enabling of Apache’s mod_disk_cache.

The Expires: and Cache-Control: headers both prevent mod_disk_cache from caching the contents.

A small patch against MediaWiki-1.16 fixes that by removing Expires: and adding s-maxage to Cache-Control:, which allows caches to serve “stale” page versions which are only mildly outdated (a few seconds).

Utilizing mod_disk_cache with MediaWiki can easily speed up the number of possible requests per-second by more than a factor of one hundred for anonymous accesses. The caching behavior in the above patch can also be enabled for logged-in users with adding this setting to MediaWiki’s LocalSettings.php:

$wgCacheLoggedInUsers = true;

I hope this helps people out there to speed up your MediaWiki installation as well, happy tuning!

]]>https://testbit.eu/using-mod_disk_cache-with-mediawiki/feed/3New Beast Websitehttps://testbit.eu/new-beast-website/
https://testbit.eu/new-beast-website/#respondSat, 05 Feb 2011 05:05:23 +0000http://blogs.testbit.eu/timj/?p=446[...]]]>Last week the Beast project went live with a new website that has been in the making since December:

The old website was several years old, adding or changing content was very cumbersome and bottlenecked on Stefan or me. All edits had to go into the source code repository, adding content pages meant editing web/Makefile.am and changing a menu entry required the entire site to be rebuilt and re-synced. Also beast.gtk.org went offline for several weeks due to hosting problems at UC Berkeley.

So in the last few weeks the Beast website has been gradually moved from beast.gtk.org to beast.testbit.eu and a different hosting service that has more resources available. In the last few years, I’ve gained experiences with Plone, Drupal, DokuWiki, Confluence, a beast-specific markup parser, Joomla, WordPress, etc. They all have their up and down sides, and while I prefer WordPress for my own blog, I’ve settled on MediaWiki for the new Beast website.

Running the new site entirely as a wiki makes the contents easily accessible for everyone willing to contribute and MediaWiki’s markup is what most people already know or are likely to learn in the future. MediaWiki must be the hardest tested collaborative editing tool available, turns out to be impressively feature rich compared to other Wiki engines, has a rich set of extensions, scripting facilities and due to Wikipedia weight a reliable maintenance future.

Much of the previously hand crafted code used for site generation and operation becomes obsolete with the migration, like the screenshot gallery PHP snippets. The entire build-from-source process can be eliminated, and running a dedicated Beast theme on MediaWiki allows editing of the menu structure in a wiki page.

Also MediaWiki allows running multiple front ends under different domains and with different themes on the same Wiki database, which allowed me to merge the Beast site and testbit.eu to reduce maintenance.

A small set of patches/extensions were used to tune MediaWiki for the site’s needs:

Enhancing the builtin search logic, so it automatically resorts to partial matches for empty result lists.

Adjusting Expires/Cache-Control headers to utilize mod_disk_cache – this increases the number of possible requests per-second by more than a factor of one hundred.

Adding support for [[local:/path/to/download/area]] URLs, to refer to downloadable files from within wiki pages.

It took a while to migrate contents gradually into MediaWiki format, as some files had to be migrated from a very old ErfurtWiki installation, some came from the source code repository and some were available in HTML only. Big Kudos to David Iberri, his online html2wiki converter (html2wiki on CPAN) has been a huge help in the process.

These days I often have a hard time to keep up with the tasks on my TODO lists, but I do manage to sneak in a spare hour here or there to look into code I authored sometime ago and that’s waiting for maintenance attention. For projects like Gtk+/GLib it’s incredibly hard to figure a good start and order for bug processing if time is sparse and the number of bugs is flooding you.

Other projects on the net use issue trackers that support user voting of individual requests, here are two examples:

Of course I do realize that we have priority and severity fields in GNOME Bugzilla, but those are for a different purpose than polling the public opinion on which bugs should be fixed best/next, or which bugs have the largest pain impact on our user base.

At least for me, a publicly open voting system for GNOME Bugs would be immensely useful to judge where it’s best to concentrate my development efforts.

Rapicorn is an Experimental UI Toolkit. Most things in a toolkit implementation will benefit from proper application of modern technologies (e.g. pthreads, XCB, Cairo, compositing, IDL, XML notation, path evaluation, DSLs, unit tests, SVG). Rapicorn is developed on this base with the aim to significantly improve developer efficiency and user experience.

This release contains a multitude of new features like Python bindings, an automated test suite, myriads of bug fixes and small improvements.

Waiting for unit compilations to finish during development, particularly with G++ and heavy optimizations can be quite time consuming. A run of make distcheck escalates the problem, because all files during a dist need to be rebuilt. Small errors triggered late in the dist build might require several time consuming restarts of the build process.

With ccache, a tool exists that can majorly speed up this process. I’ve been using it successfully for several years now. A related development aid is colorgcc which is a colorization wrapper for gcc. This also is under heavy use for my development. It turns out that some tweaks are needed to get it working together with ccache though.

Here’s how to use the tools in combination to speed up regular project builds and also distcheck:

Get and install ccache.
Wrapping compiler invocations with ccache will cache recompilations of the same units. ccache needs a directory to store the cache contents; this defaults to ~/.ccache which might not be the best choice for network homes or encrypted homes. Using /tmp is also suboptimal since it is usually cleaned upon reboots. So I’d recommend to put this into your .bashrc:

export CCACHE_DIR="/var/tmp/ccache-$USER"

This will retain the compiler cache across reboots in /var/tmp.

Get and install colorgcc.
Then some patching is in order, especially if you use multiple versions of gcc and need ccache integration. Based on colorgcc from Ubuntu-9.10, I’ve prepared a patch for this (it has been sent to Jamie Moyers some while ago) to be found here: http://testbit.eu/~timj/blogstuff/xyv-colorgcc.diffTo briefly summarize its effects:

This allows colorgcc to be used through a bunch of links, e.g. by creating:

Doxer is a documentation tool initiated originally for generating source code documentation. This is the first public release of Doxer as a separate package (it has been included in other packages previously).

This release contains a wiki markup parser, a corresponding HTML generator, and a Drupal input format module. The markup syntax is designed to cover the feature set required to write source code documentation and general purpose documents. The parser and HTML generator have a strong focus on robustness to support the full range of user sophistication found on general purpose websites. An extensive test suite accompanies the development.

Lots and lots of things have been going on around me lately, but that’s best left for other posts if I ever get around to do them.

A few months ago, I’ve sat down with quite some help by others and collected the input and feedback around Gtk+ 3.0. The outcome of that was a first Gtk+ 3 Roadmap draft that was sent around to the core team.

I’d like to thank everyone who participated in the fruitful discussions leading to this and particularly Stormy and Dave Neary for their suggestions on the post-draft process.

Cody Russell has kindly volunteered to wikify the roadmap, so future alterations will be easy. I much appreciate his initiative, especially because I can’t foresee to have much time around the roadmap personally in the near future.

Managing bug lists has become an ubiquitous task when dealing with the GNOME or Nokia bugzillas. At some point I became fed up with the involved cut and pasting, searching and sorting, so I cooked up a small command line utility to construct bug list URLs and format bug list summaries in text form. Here’s it in action:

It knows a good number of bugzillas, such as the ones from Gnome, FreeDesktop, Maemo, Nokia, OpenedHand, GCC, LibC and Mozilla. More bugzilla URLs can easily be added, and it handles HTTPS authentication that some of the corporate bugzilla installations require.

Two weeks ago, Bruce Byfield did an interview with me about Manju. The article is now up at linux.com and gives a general overview of the project scope: Manju Project Article.

There’s also been a German interview with Sven Herzberg and me some months back at the Berlin Hackfest, which covers some historical background around Gtk+, some technical bits, possible future directions and how it relates to GNOME. The German podcast of this is up at Chaosradio Express: GTK+ und GNOME Podcast (German podcast).

Andreas Nilsson and I recently started the Manju project which aims at creating graphical widget toolkit themes from SVG files in a toolkit independent fashion.

It combines the idea of a pixmap theme engine, Jimmac’s “One Canvas Workflow”, SVG markup features and Inkscape export functionality to fully automate the creation of a scalable and stretchable pixmap theme from an SVG source.

Guadec has been in interesting conference, particularly because it took place in Istanbul this year. I tried to keep a few notes throughout the days to wrap up the experience and discussions here.

Sunday

Headed off for Istanbul, partial Imendio meet-up at the airport in Vienna, gathered remaining Imendians at the airport in Istanbul. Like many others, we stayed at the Golden Long Hotel near the seaside.

Monday

Kris and I met up with the Gnome release team where we summarized the Gtk+-3.0 ideas that have been cooked up during and after the Berlin Gtk+ Hackfest.

Tuesday

I was planning to attend the Maemo BOF in the morning, but shortly after arriving at the conference venue, Federico literally dragged me into the DVCS BOF. Still, I only managed to attend the second half of it which was almost exclusively about Bazaar features. People told me the first hour had been quite the contrary and focused mostly on Git hyping. Clearly, there was no consensus after the meeting on what the future versioning system in Gnome will be.
I’m not surprised that is the case. As things currently stand, SVN has very active development and user communities, Git is very actively developed, Bazaar is as well, as are a couple other VCSes. Active developer and user communities are generally a good sign for a healthy project and also an indicator for future relevance. Thus, in any larger community such as the Gnome community, it’s easy to find lots of critics and lots of supporters for each of the bigger versioning systems and that’s unlikely to change much. Consequently, there’ll not be an easy consensus on switching to a single versioning system any time soon, so I think the most productive approach for Gnome to take is to prepare the hosting of multiple VCSes, certainly SVN, Git and Bazaar. Needless to say that cross-VCS integration will also become increasingly important in the future, so focus on maturing and extending git-svn, bzr-svn, bzr-git and the like makes a lot of sense.

Later that day, we had a Gtk+ Developers meeting in the medium sized presentation room. The place was a bit too large to have the planned face to face discussions so we’ve had to sacrifice some of the spontaneity to microphone resource sharing. Kris took minutes during the meeting and will probably post those to gtk-devel-list once he’s found a minute to process them. The meeting was quite productive nevertheless, we discussed the upcoming Gtk+ 2.14, features and schedule for 2.16 and 3.0 and a bit of the post-3.0 road map.
Right after that discussion, Kris and I attended the advisory board meeting where we briefly wrapped up the developers meeting, the Gtk+ Hackfest in Berlin and the improvements the Gtk+ project has seen since the State of the Gtk+ Maintenance email. In particular, we stressed that we now have the GtkTasks and Gtk+ 3.0 Tasks wiki pages which can serve as an entry point for contributors and assistants to the project at various experience levels, in particular for companies that want to sponsor developer resources. Also for people that have an interest in long term Gtk+ project involvement, feel free to read up on how to become a Gtk+ maintainer.

Pizza looks weird in Istanbul BTW:

Wednesday

Almost by accident (I was mostly looking for an air conditioned hall in the afternoon), I happened to be watching “Gnome Documentation: A year in review” by Don Scorgie where he described the new user documentation tool Mallard. For some time now, I’ve been working on a Wiki syntax parser for Doxer to unify the markup I have to use for my blog, inline documentation and CMS content markup at testbit.eu. (I have a blog entry in the queue on this for another day.) In this context, implementing a Doxer markup backend that generates Mallard’s XML input could be an attractive markup alternative for future Gnome documentation – it at least ended up on my ever growing TODO list.

Lots of people approached me throughout this and the following days for a chat about Gtk+-3.0 and what comes after that. With the merging of the GSEAL branch into upstream trunk recently, there’s been a lot of focus on the technical preparative work we’re doing for the actual 3.0 release which is planned as an ABI break to Gtk+-2.16 without adding any new features (it’s basically just a re-release with all deprecated code removed and current “private” API really made private by moving it to non-installed source files).

What has been lacking emphasis in this course is that 3.0 is going to be the necessary enabler, needed to work on implementing future visions of Gtk+ and to refactor the code base back to a healthy state where it becomes maintainable again. Quite expectedly, the need for the sealing and accompanied ABI break has been questioned several times, so I’ll reiterate the reasoning here:

Everybody falls the first time.
Code development in open source projects is very evolutionary, especially for projects that don’t clone or reimplement existing specified APIs like Libc. A whole chapter is spent on prototypes in The Mythical Man-Month: “Plan to throw one away; you will, anyhow.”
So newly added components and APIs are almost certain to need fixups or revamps in future iterations (likely more than once). If critical internals are being exposed and eternal ABI stability has been promised, this however becomes impossible. Given the development history of Gtk+ and the variety of interests in this project, it is vital for its future success to prepare for future changes and allow iterative improvements. After all, progressive improvements, appreciation of contributions and adaptions to changing circumstances is where the free software development model shows its strength. The waterfall design model is not it.

Gtk+-2.x is essentially a dead end.
Everybody agrees that Gtk+-2.x is pretty much dead in a few revisions because of the huge work involved in its maintenance and no relief in sight with its current ABI maintenance policy. This is at least true for all current and
past core team members (i.e. everyone who actively tried maintaining 2.x over a significant period). The question is whether we move to an entirely new toolkit (Clutter, Rapicorn, Qt, HippoCanvas, etc) or whether that “new” toolkit is Gtk+-3.0 which may be largely API compatible with Gtk+-2.x. In either case, applications and libraries will need to migrate to a new toolkit base with a different ABI, the main difference is the involved porting effort.

GLib and Gtk+ do have a means to deal with API changes:
1) We provide new alternative interfaces (functions).
2) We deprecate old interfaces (functions) or provide compatibility code in old interfaces.
Notably, this does only work for API that is exported via function symbols. Structure fields that are directly accessed from application code can’t be deprecated and removed without breaking ABI, and there is no compatibility code upstream could provide for these kind of accesses either. That’s why we want to move away from exposing any structure internals in 3.0 and beyond.

3.0 will ABI-incompatibly remove all deprecated and private APIs.
Of course, the above described deprecation scheme only scales well if deprecated APIs are really removed from the code base at some point. Technically, this is an ABI break which is why GLib/Gtk+ have not been doing this since 2.0. However, lots of other vendors do this to keep a healthy code base, e.g. Qt does break ABI between major releases, Python 3.0 will be incompatible with 2.5, Apple does remove long deprecated APIs in newer releases of Mac OS X, Symbian broke API and ABI in 9.x, Microsoft broke behavior from .NET 1.1 to 2.0, and the list goes on…
By exposing only function symbols as future public interfaces, we’ll be able to provide arbitrary compatibility functionality for old interfaces on top of new components, add helpful runtime warnings for iterative migration and constrain future ABI breaks to removal of properly deprecated interfaces.

User visible gains are post-3.0 features:
Since GLib and Gtk+ are largely volunteer contribution based projects, it’s close to impossible to plan exact arrival of future features. However the following is a list of things that have (partially) been discussed as post-3.0 work during the Gtk+ Hackfest already:

Full support of alpha transparency for all widgets;

Support for (partial) stacking of widgets (needs transparency);

Offering easier layouting facilities;

Support for animated visible transitions between widget states;

Providing new UI metaphors based on simulation of physical effects like acceleration, 3D browsing of image collections, 3D skimming through notebook pages, and more;

Using IDL based type data generators and code generators to improve the way widgets are implemented;

Implementing a new theming system for the toolkit;

Moving towards exposing widget features only via interfaces that have their own handle (asymmetric query_interface).

This is how GSEAL, the Gtk+-3.0 release and a couple remaining outstanding tasks are going to enable development of exciting future user visible features. The next step for Gtk+ to get work in visionary areas off the ground is to start consideration of feature feasibility and implementations, required resources and tentative schedules.

At the end of the day, we had the Opening Cocktails Party, during which I managed to catch Hallski tattooing J5:

Thursday

Thursdays most interesting event was of course the keynote by Kris which got hijacked by the Gnome release team for the announcement of Gnome 3.0 which is essentially Gnome 2.30 cleaned up and based on Gtk+-3.x. Kris’ slides are available online:

The slides provide a good overview of what Gtk+-2.14 will bring, prospects for 2.16 and visions/requests from the community for Gtk+’s future. As previously described, Gtk+-3.0 is about enabling refactorings and development of new features, and the plan is to do our best to make the transition away from old deprecated code as easy as possible. Other than properly porting an existing Gtk+-2.x application to work with the G_DISABLE_DEPRECATED, GTK_DISABLE_DEPRECATED, GSEAL_ENABLE switches, no additional changes will be required to build and run an application on Gtk+-3.0.

At the end of the day, there was the boat trip through the Bosporous which provided a beautiful sight along the coast line.

Friday

I managed to attend the latter half of the lightning talks which was as always quite interesting. I should probably ignore my laziness and actually prepare short lightning talks for next year about Rapicorn and possibly Doxer…

I took particular interest in Transifex, an online translation platform that can work together with multiple VCSes and that we’d ideally move all Gnome translations to in the future. There are two things I’d like to see fixed in a future translation workflow from a developer perspective:

The .po templates should really be generated by the developers of the upstream project by automatic means, e.g.: make update-po -C po/
So the upstream version of intltool and po/Makefile.* are used instead of possibly broken or outdated intltool/gettext versions on the translators system.

Developers should be able to determine merge points for translations, and also review related non-po file changes, rather than having translators wildly commit into upstream repositories (which may conflict with other VCS workflows like branch merges or commits around release phases).

Later on, Federico presented his ideas for timeline tabs for the desktop. This should make it rather easy to find documents or URLs from previous days or weeks, because out of natural necessity, humans generally have good chronological associations. So the new and nice part about this approach is that it can provide good visual access to the chronologic dimension, something a file system doesn’t usually reveal easily, and that’s not easily made accessible by the most prominent desktop metaphors either.

I feel very tempted to start an implementation of the desktop tabs with Rapicorn, however with a few refinements of Federico’s proposal:

The view should provide a “chronological zoom” slider to switch the view between years/months/weeks/days/hours.

To be most useful, we’ll need a crawler that tries to (re-)construct past file modification history without relying on programs pushing journal entries about file edits. This will be needed anyway unless every program on this planet provides file editing journal information.

I think the journaling hooks need to be implemented via DBus and not rely on Nautilus, so they’re usable by all cross-desktop applications and non-GUI programs.

Various filters by file extensions, magic and possibly more will also be needed in the tab view (this was partly raised during the discussion phase at the end of the presentation).

Saturday

About half of the Imendians headed home on Saturday, we had some early leaves on Friday already and left some others in Istanbul for additional vacations.

Oh, and since I’ve been asked about my nickname here and there, I decided to add tabs to my hackergotchi for clarification:

Aftermath

There have been quite some discussions on the Gtk+-3.0 plan after Kris’ keynote. One thing that was brought up is that releasing an ABI incompatible but featureless new version of Gtk+ and calling it 3.0 is rather unconventional. An alternative scheme could involve releasing the ABI incompatible cleaned up version as 2.99.0, make 2.99.x the new development branch and release 3.0 with cleaned up ABI and new features (would have been 3.2 in the original plan).

Tests that abort, e.g. via g_assert() or g_error(), are registered as failing tests with the framework. Also, the gtester utility used to implement the above Makefile rules will restart a test binary after a test function failed and continue to run remaining tests if g_test_add_func() has been used multiple times.

Checks in tests can be written with if() and g_error() or exit(1), or simply by using variants of g_assert(). For unit tests in particular, an extended set of assertions has been added, the benefit of using these are the printouts of the involved values when an assertion doesn’t hold:

There’s of course lots of room left to improve GLib and Gtk+ unit tests, and also to improve the current framework. For a speculative, non-comprehensive list, here are some ideas from the unit testing section of my personal TODO:

Publically install a variant of the Makefile.decl file used in Gtk+ to implement the test framework rules and Xvfb swallowing of test programs. This is needed by other projects to run unit tests the way Gtk+ does.

The XML GUI definition files used in Rapicorn and also in Beast (described briefly in an earlier blog post) supported a simple $(function,arguments...) evaluation syntax, similar to GNU Make. I’ve never been very happy with this syntax, but it was fairly easy to implement at the start and followed naturally from early $VARIABLE expansion features. At some point last year I considered various alternative syntax variants and discussed the ideas with Stefan Westerfeld. Over the course of the last two months, I finally got around to implement them.

I’ve never grown familiar with reverse polish notation, and although Guile is the canonical scripting language for Beast, I’ve always found myself very inefficient with expressing my thoughts in lisp expressions. So the new syntax definitely had to support infix expressions – despite the more complex parsing logic required to parse them. Bison already ships with an example calculator that parses infix expressions, so that’s a quick start as far as the syntax rules go. Integration is quite a different story though, more on that later.

Since in Rapicorn the expressions are used to compute widget property values, they are likely to be executed very often, i.e. each time a widget is created from an XML file description. Consequently, I wanted to use a pre-parsed AST to carry out the evaluation and avoid mixing evaluation logic with parser logic, which would have forced reparsing expressions upon each evaluation. At first I quickly threw together some C++ classes for the arithmetic operators and used those as nodes for the AST, similar to:

In this notation, FloatingPoint includes hexadecimal numbers and of course integers and the quoted strings support C style escape sequences like octal numbers, ‘\n‘, ‘\t‘ and so on. The functions can be implemented by the parser API user, but a good set of standard arithmetic functions is already supported like ceil(), floor(), min(), max(), log(), etc. So it’s a very basic parser, but covers the vast majority of expressions needed to position widgets or configure packing properties.

One thing that turned out to be tricky is the binary operator semantics for strings. At the very least, I wanted support for "string" + "string" and "string" == "string". Since both operators are supported for numbers as well, the exact behavior of "string" + FloatingPoint and "string" == FloatingPoint had to be defined. I managed to find a few programming language precedents here in Perl, Python, and ECMAScript (Javascript). They of course all handle the cases differently. In the end I settled with ECMAScript semantics:

Value1 == Value2 # does string comparisons if both values are strings
Value1 + Value2 # does string conversion if either value is a string

Unit testing for the parser turned out to be particularly easy to implement. All that’s needed is a small utility that reads expressions and prints/verifies the evaluation results. Throwing in some additional shell code allowed a normal text file to drive unit testing. It simply contains expressions and expected results on alternating lines. Btw, libreadline can be really handy in cases like this. Using it takes a mere 5-10 lines of additional code to support a nice interactive command line interface including history for the evaluator test shell.

After some initial testing, the C++ AST node classes seemed like an awful lot of pointers, fragmentation and VTable calls for a supposedly straight forward expression evaluation. Also, adding the missing destruction semantics would have significantly increased the existing class logic. So I tried to come up with a leaner and foremost flat memory presentation. In the end, I settled with a single array that grows in 4 byte (integer) steps, embeds strings literally (padded to 4 byte alignment) and uses array offsets instead of pointers for references. A single multiplication operator is encoded with 3 integers that way: MUL_opcode factor1_index factor2_index. That’s essentially 12 bytes per binary operator, still significantly more than the parser input, but also significantly smaller than allocating an equivalent C++ class. Evaluation of the expression stored in the array doesn’t need any VTable calls, and a single straight forward evaluation function can be used, that implements the different operators as switch statement cases. Also release semantics are persuasively trivial for a single consecutive array.

What’s left was to figure a way to embed expression evaluation in XML values or text blocks. Previously, $(expression) was substituted everywhere, but with the new syntax, variables (or rather constants defined within the Rapicorn core or via the ‘‘ syntax supported by Rapicorn XML files) didn’t use a $-prefix anymore. So sticking with $() seemed to make little sense. As it’s implemented now, backticks are used to cause expression evaluation, with the empty expression evaluating to a single backtick:

We will see how useful the current expression style turns out to be. I don’t consider every implementation bit outlined here solidly engraved into stone yet. So as always, I’m open to constructive feedback.

As forewarned, I have a few more words on integrating Flex and Bison with each other and into one’s own library. First, Flex and Bison turned out not to be exactly simple to configure, especially if none of the generated symbols should be exported from a library or clash with a possible second parser linked into the same library or program. Also some fiddling is required to pass on proper line count information from the lexer to the parser, getting character counts as well is non-trivial but lucky wasn’t strictly needed for Rapicorn expressions. The simplest setup I managed to come up with works as follows:

The only compiled file in this list is sinfex.cc which includes sinfex.lgen and sinfex.ygen as part of an anonymous C++ namespace. A linker script ldscript.map used when finally linking the library prevents anonymous symbols from being exported. The anonymous namespacing of everything declared in sinfex.lgen and sinfex.ygen is what prevents clashes with a possible second parser linked into the library. This isn’t as elegant as i was hoping for, but at least effective in a practical sense. There unfortunately is no way to configure Flex or Bison to generate only static functions and variables. And yes, I have also looked into variants like flex++, bison++, bisonc++, byacc, etc, but they usually show much of the same problems and also tend to make matters worse by generating more files or more complex code.

Due to a sports accident, our booth bunny Sven Herzberg unfortunately couldn’t make it, so on Tuesday I took over booth management and merchandise from him and hurried to Berlin in an ICE instead of a car as was originally planned. In Berlin, I met up with Mathias Hasselmann who brought the European Gnome event box and together we set up the booth until late in the night.

On Wednesday morning, Michael Köchling and Christian Dywan arrived, so we had enough people to properly man the booth. Michael seems to be an early riser, since he managed to show up at 09:00 for all days, so i passed the booth keys on to him.

Around lunch time, I was dragged away for an interview about business involvement in free software and Gnome in particular by Malgorzata Ciesielska, a business school student from Copenhagen. She also interviewed other people like Lennart Poettering who also sporadically hung around our booth.

Later that day, Lennart and i went over the new libcanberra API in a lengthy discussion. Libcanberra is a new library for playback or activation of sound events in response to desktop actions that Lennart currently works on. We talked about the needs of timing information for some usage cases, possibly also dispatching forced feedback controls via the library and implementation of a Gtk+ module to hook canberra functionality up with GUI events. What turned out to be a bit tricky is to derive actual semantic information from the low level X event notification that Gtk+ signals proxy, such as dialog-confirmed, dialog-cancelled, menu-item-selected, menu-item-cancelled, combobox-popup, combobox-selected, combobox-cancelled, etc. This extraction requires significantly more logic and special casing of event notification than Lennart apparently had originally hoped for.

On Thursday I attended the Linux Kernel – Quo vadis? talk by Thomas Gleixner which was quite interesting. I managed to catch him afterwards to talk about the prospects of having a memory pressure signal in the Linux kernel. For GLib and Gtk+, this’d be quite useful to voluntarily release pixmap or GSlice caches, particularly desired on embedded platforms.

Friday I sat down with Vincent Untz for a very productive discussion about Gnome/Gtk+ release prospects, autotools, intltool features and more. Later I had a chance to chat with Alexander Neundorf about KDE’s recent CMake migration process. Overall, they seem to be pretty happy with the results. Major benefits from migrating from autotools to CMake seem to be:

Build process speedups due to getting rid of libtool.

Simplicity; the build setup is back to a manageable level again. For KDE, the previous combinatoric mess of autotools was hardly fully understood by any single person.

Unification/merging of build files for Unixes and Windows. (Duplication of build logic between auto* files, nmake and MS project files is currently a major annoyance for Gtk+’s Win32 maintainers.)

After that, I went to Jono Bacon‘s talk in which he explained how the free software community is a creative and productive community, which sets it apart from other common community types in our society (usually fan communities). He went on pointing out how this collaborative and open community as a whole (and thus every significant contribution to it) impacts and gradually changes the really big IT companies from within in an unprecedented manner. Come to think of it, this is an incredible achievement that we should rejoice in, especially because it is a morally correct change in that it strives towards openness.

All in all, it was a nice conference again. Personally, I particularly enjoy meeting up with other hackers for productive face to face sync ups. So I’d like to thank Christian and especially Michael for their great efforts in patiently answering bypasser’s Gnome questions at the booth, while I wandered off to talks or had technical discussions in its back.

Amongst many other things during the Gtk+ Hackfest 2008, it was brought to my attention that Gtk+ maintainership is sometimes perceived as some kind of leet circle, hard to join for newcomers. I can’t really imagine how “hard” it is for newcomers to join Gtk+ maintenance these days. The learning curve probably is steeper now than it was in the last decade, but then, we do have documentation now and didn’t have any back then…

In any case, I think that definitely none of the Gtk+/GLib core maintainers would want to bar someone else from contributions or helping out with maintenance tasks. So to aid that process, I’ve written down what I keep telling people who approach me about this in person. A lot of it might seem overly obvious to veterans, but for newcomers or contributors looking into extending their activities on the project I hope to provide helpful starting points.

Much of Gtk+ is still maintained as a huge monolithic whole. That is, a very few people are taking care of a wide variety of components. I think ultimately we would all be much better off, if we had more people being responsible for individual components like particular widgets (e.g. GtkNotebook) or widget families (e.g. GtkBox, GtkHBox, GtkVBox). So the following are steps and tasks useful for anyone wanting to get into Gtk+ component maintenance.

First of all, we have the GtkTasks wiki page, a task list with various ways in which people can contribute to the Gtk+ project and start getting involved. In particular for the following positions, no one has signed up so far to volunteer on a regular basis:

Patch/Bug Monitoring & Filing – collect bugs from mailing lists and IRC to make sure they are being filed.

FAQ Maintainer – this involves monitoring the mailing list(s), taking CC:-mails from other people on possible FAQ material and regularly updating the FAQ accordingly.

Build Monitoring – run and maintain Gtk+ tool-chain builds on various platforms.

Making Releases – we are looking for someone assisting in the release process and to take over release building during some periods in the future.

Implementing Test Programs – there’s much work needed in our unit test coverage, so we’re always looking for people willing to implement more unit tests.

For general information about the maintenance situation, the State of the Gtk+ Maintenance write-up is still fairly valid. However many good things have been suggested by the community since, such as the GtkTasks page and the Hackfest. At the end, the write-up addresses how occasional contributors or developers at different experience levels can help out the project. For instance with activities such as: Bug triage and verification, Review and clarify documentation, Revision hunting for regressions, Refactor code areas, Work on optimizations, and the list goes on.

If none of this sounds interesting to potential new maintainers, be warned that regular maintenance means having to put up with pretty much all of these at some point.
However, probably the most straight forward way to take on maintenance for a particular code portion (usually a particular widget) is to start becoming familiar with it and work on improving it right away:

Cleanup indentation – lots of contributions to Gtk+ in the past have weakened coding style in various areas. In general we use GNU coding style plus a few extra rules similar to the Gimp coding style.

Perform code cleanups – look for outstanding migrations/refactorings in the code or if the implementation could be simplified/streamlined in areas.

Check documentation and examples – contributing by improving the existing documentation, testing existing examples and providing new ones is a very straight forward way to learn about a new component. Also, documentation patches are usually easily and quickly approved.

Provide unit tests – writing new unit tests for existing component APIs is even better than providing documentation examples. You get immediate feedback and they should in general be easy to approve to go into upstream. Also, it is definitely an area where Gtk+ and GLib still need lots of work.

Review bug reports and patches – go through the Gtk+ bug load of a particular component, see what issues could be closed or need info. Find patches that could be applied or need work and provide/fix patches along the way. Also, feel free to provide review for existing patches where you feel confident to provide reasonable input. For existing maintainers, looking at other people’s review is one of the best ways to figure if another person is up to the task of taking over component maintenance.

Actively nag existing maintainers or people with SVN accounts for trivial patches (like Cody and Mathias who signed up for Patch testing & committing) to review, approve and apply your changes.

The first few points actually mean working with the code by providing patches, and filing new bug reports with those patches attached. While this may at first increase our bug load, if someone shows sincere interest in taking over component maintenance, sticks to the coding style and provides useful patches, there’s nothing stopping us from granting new SVN accounts so contributors can commit their own patches after approval.

Finally, the project is welcoming every new contributor. But be reminded that no approval is needed from anyone to start your work. In actuality, asking for it will most probably get you a reserved or negative answer, because improving the project is all about working on it, not making or requesting promises. So, everyone is invited to read through the task lists/descriptions and get their hands dirty immediately. A fair bit of self incentive is needed to take on maintenance of a component anyway, so you’ll have to get yourself motivated on your own there won’t be anyone else doing it for you.

Lots of things have happened since the last snapshot over a year ago, some of which kept me from making releases or snapshots earlier.
Others actually took place in the code base as interesting developments, summarized below.

There’s quite some more stuff in my development queue, but at some point one just has to draw a line and throw out what’s vaguely ready so far, so this is it for today:

There have been several requests about hosting Gtk+ (and GLib) as a Git repository recently and since that topic has come up more and more often, I meant to write about this for quite some time.

Let’s first take a look at the actual motivation for such a move. There are a good number of advantages we would get out of using Git as a source code repository for Gtk+ and GLib:

We can work offline with the Gtk+ history and source code.

We can work much faster on Gtk+ even when online with tools such as git-log and git-blame.

It will be much easier for people to branch off and do development on their own local branches for a while, exchange their code easily with pulling branches between private repositories, etc. I.e. get all the advantages of a truly distributed versioning system.

We can make proper public backups of the source code repositories. This ought to be possible already via svnsync, but isn’t for svn.gnome.org because we run an svn 1.3.x server instead of an svn 1.4.x server that is required by svnsync. (Yes, this issue has been raised with the sysadmins already.)

A quick poll on IRC has shown that all affected core maintainers are pretty much favoring Git over SVN as a source code repository for GLib/Gtk+.

However, here are the points to be considered for not moving the repositories to Git (just yet):

Complexity; Git is often perceived to be harder to use than SVN, there are several attempts to mitigate this problem though: Easy Git Giggle yyhelp.
With some of the above, Git is as easy to use as SVN is, so Git complexity doesn’t need to be holding anyone off at this point.

Git may be stable and mature these days, but git-svn is not there yet. It is generally good enough to create local Git mirrors of SVN repositories and work from those to have most of the Git convenience on top of an SVN project.
But git-svn has seen structural changes recently, quite some rewriting and bug fixing that indicate it’s still too much in flux for the one-and-only SVN->Git migration for projects at the scale of Gtk+. This is not meant as criticism on git-svn, fairly the opposite actually. It’s good to see such an important component be alive and vivid. All issues I’ve raised with the maintainer so far have been addressed, but it seems advisable to wait for some stabilization before trusting all the Gtk+ history to it.

Gitweb interfaces already exist for GLib/Gtk+ mirrors, for example on testbit: Testbit Git Browser.
These can be used for cloning which is much faster than a full git-svn import. Alternatively, shallow git-svn imports can be created like this:

This will create a repository with a mere ~1000 revisions, including all changes since we branched for 2.12. We’re using such a shallow repository for faster cloning of our GSEAL() development at Imendio: view gtk+.git.

In summer 2006, we’ve had the first test migration of all of GNOME CVS to SVN, in December 2006 we’ve had the final migration. During that period, the Beast project stayed migrated to SVN to work out the quirks from the CVS->SVN migration before we migrate all other GNOME modules and have to fix up everyones converted modules. There were quite some issues that needed fixing after the initial test migration and in the end we had to rebuild the Beast development history from pieces. Preventing the other GNOME modules from such hassle was the entire point in migrating Beast early on, so I’m not complaining.
However, given the size and importance of GLib/Gtk+, the development history of those projects shouldn’t be put at a similar risk. That is, GLib/Gtk+ shouldn’t be pioneering our next source repository migration, let some other small project do this and work out the quirks first.

git-svn already provides a good part of the Git advantages to developers while Gtk+ stays hosted in SVN. Albeit due to mismatching hashes, syncing branches between distinct git-svn checkouts of different people is tedious. Setting up an “official” git-svn mirror on git.gtk.org could probably help here. Also, to ease integration, jhbuild could be extended to use git-svn instead of svn commandos to update SVN modules, if the current module has a .git/ subdirectory instead of a .svn/ subdirectory.

The bottom line is, there’s a good number of advantages that Git already can (or could) provide for our development even without migrating the repositories to Git right away. When exactly will be a good point for migrating GLib/Gtk+ and possibly other GNOME modules might not be an easy call, but right now is definitely too early.

]]>https://testbit.eu/05022008-thread-safe-class-initializers/feed/130.10.2007 YummyYummySourceControl Version 0.9https://testbit.eu/30102007-yummyyummysourcecontrol-version-09/
https://testbit.eu/30102007-yummyyummysourcecontrol-version-09/#commentsTue, 30 Oct 2007 09:54:04 +0000http://blogs.gnome.org/timj/2007/10/30/30102007-yummyyummysourcecontrol-version-09/[...]]]>A couple people have reported minor and major bugs in the last yyhelp version, particularly after yycommit got reimplemented to operate on top of git-commit(1) instead of cg-commit(1). Besides some others, this new release fixes all known yycommit issues and also (re-)introduces some new features: yyhelp (v0.9)

Overview of Changes in YummyYummySourceControl-0.9:
* use plain "git-commit" with temporary index file to stage
commit files, this works around git-commit-1.5.2.5 not
handling deleted files as command line args correctly.
* also list remote branches for yylsbranches.
* fix leading dot getting stripped from modified files if $gitprefix=.
* require and use gawk for time formatting, which mawk doesn't support.
* properly honor the [FILES...] arguments to yycommit.
* terminate sed command blocks with semicolon (needed on BSD).
* resurrected yyhelp.auto-push-commits functionality of yycommit.

This version supports a new command to grep and match an extended regular expression on the full project history by invoking git-grep(1) on all existing commits, displaying matches by commit hash and file name. Also, a CVS/SVN/Cogito style version of yycommit has now been implemented on top of git-commit(1). So YummyYummySourceControl-0.8 finally gets rid of the last remaining cogito dependency, making it a free standing Git Porcelain Suite.
Of course there have also been some other miscellaneous changes and docu updates.

I have been asked some of the purpose of YummyYummySourceControl, so i will extend a bit on that here:

– On some occasions i had to dig quite deeply into the nitty-gritty details of git (and cogito). However i can simply not be bothered to deal with the complexity of its command set for everyday use. I need my source control interface to be really simple when i concentrate on the various projects i’m involved in.

– I refuse to bother with the state (or existence) of git’s index file for anything but the correct implementation of yyhelp. SVN and CVS don’t force me to deal with an intermediate cache state between repository and working tree, YummyYummySourceControl keeps me from seeing it in git.

– I might be using yydiff and yyview up to a dozen or so times per hour while working. Passing options along here or piping the output to a pager or starting stuff in the background really becomes unaffordable at that rate. These commands do all the necessary stuff out of the box, all i have to type is “yyd”+Tab+Return and “yyv”+Tab+Return for each of them now.

– I’m too old and stupid to remember to pull after i’ve pushed and what the git-svn(1) command variants look like. yypull and yypushpull handle that for me.

– Personally, i find ChangeLog style logs much more parsable than git-log(1) output. yyChangeLog gives me that for the git commit history.

– To allow proper git merging, cherry picking, rebasing, etc, i’ve switched to commit my ChangeLog entries as commit message only (with introductionary one-liner instead of the date + email header) and for Rapicorn auto-generate the project ChangeLog for tarballs. Even if the ChangeLog is not auto generated as with Beast or Gtk+, i can still use this commit message scheme for git-svn checkouts, as long as i update the project’s SVN ChangeLog before yypushpull-ing the changes from my local tree to SVN. yyChangeLog -s will generate the remaining ChangeLog portion missing from SVN.

– To help against sudden confusion and lengthy URL searches, yyinfo tells me all i want to know about a repository (git URL + revision, SVN URL + revision, etc).

And i love to have grep-able access to all the data involved in my day to day activities, yyHistoryGrep now gives me that for my project histories.

Given git’s speed, the possibility for light-weight local repositories, and using yyhelp’s ease, revision control has become so available for me, that i could very comfortably apply it to portions of my home directory and /etc/ on a remote server (e.g. i’ve created a ~/.git/ a couple months ago where i checked in a handful selected files like TODO, my blog diary, .emacs and a few other frequently changing TODO-style files and i’m happily committing to it multiple times a day).

I’m not claiming that yyhelp solves anyone else’s problems, but it certainly has helped my SCM usage a great deal, and for me serves as a key to make a frequently used subset of the git machinery accessible at very few finger tips.

]]>https://testbit.eu/13102007-yummy-yummy-porcelain-version-08/feed/121.09.2007 Beasty Bits (final spurt)https://testbit.eu/21092007-beasty-bits-final-spurt/
https://testbit.eu/21092007-beasty-bits-final-spurt/#commentsFri, 21 Sep 2007 22:09:38 +0000http://blogs.gnome.org/timj/2007/09/21/21092007-beasty-bits-final-spurt/[...]]]>We’ve been fairly busy recently with resolving the milestone bugs of the next Beast release. The good news is that pretty much all of the hard issues are sorted out by now, the bad news is that according to the release plan some essential release features are still missing

It’s the large recent work that Stefan Westerfeld (who btw finally decided to start his own blog) has put into shaping up the remaining bits of bsewavetool (man page draft).
This is a very handy tool, that’ll finally be installed for public consumption in the upcoming release (it has been developed in SVN since 2004!). It can clip/normalize/ogg-encode/highpass/lowpass/upsample/downsample/etc chunks from the BSE multi sample files, and is normally used for shell-scripting the construction of sample kits from an unsorted pile of raw unprocessed sample data (note to self: blog some example use cases after the release).

Oh – and before i forget, the synthesis link section on the website got some new updates as well: Beast Synthesis Links,

Also, Stefan pointed me at a YouTube video of Beast the other day:

The video is very nicely done, but it has the usual YouTubeish low quality artefacts.

]]>https://testbit.eu/21092007-beasty-bits-final-spurt/feed/219.09.2007 Yummy-Yummy git-svnhttps://testbit.eu/19092007-yummy-yummy-git-svn/
https://testbit.eu/19092007-yummy-yummy-git-svn/#commentsWed, 19 Sep 2007 13:30:44 +0000http://blogs.gnome.org/timj/2007/09/19/19092007-yummy-yummy-git-svn/[...]]]>I’ve had to add another bit to yyhelp to make my everyday coding more convenient, so it got support for git-svn(1). The yypull and yypushpull commands will now detect a git(7) repository set up with git-svn, and unless the repository also has a regular git upstream, pulling and pushing will be carried out via git-svn rebase and git-svn dcommit. yyinfo shows the update method being used for a repository and also displays the SVN HEAD revision now if any. That, plus miscellaneous changes make version 0.6 of yyhelp:

Overview of Changes in YummyYummySourceControl-0.6:
* for git-svn repositories, use rebase and dcommit to pull and push
* fixed argument parsing for yybranch and yybranchdel
* support revision argument for yydiff
* check revision argument to yywarp

]]>https://testbit.eu/19092007-yummy-yummy-git-svn/feed/108.09.2007 Yummy-Yummy Bugletshttps://testbit.eu/08092007-yummy-yummy-buglets/
https://testbit.eu/08092007-yummy-yummy-buglets/#commentsSat, 08 Sep 2007 16:41:31 +0000http://blogs.gnome.org/timj/2007/09/08/08092007-yummy-yummy-buglets/[...]]]>The YummyYummySourceControl script got a few buglet fixups during the last days, so I’ve put up version 0.5b: yyhelp.
Here’s the obligatory list of changes since the last version:

]]>https://testbit.eu/08092007-yummy-yummy-buglets/feed/129.08.2007 Yummy-Yummy Againhttps://testbit.eu/29082007-yummy-yummy-again/
https://testbit.eu/29082007-yummy-yummy-again/#commentsWed, 29 Aug 2007 14:58:23 +0000http://blogs.gnome.org/timj/2007/08/29/29082007-yummy-yummy-again/[...]]]>I’ve put up a new release of the slightly renamed YummyYummySourceControl script here: yyhelp (0.5). The script still depends on cogito(7) for the implementation of yycommit, but other such dependencies have been removed at this point so it’ll eventually depend on Git(7) only. Following is a sum up of the changes between the last version and v0.5:

]]>https://testbit.eu/29082007-yummy-yummy-again/feed/217.08.2007 What are ELF libraries good for anyway?https://testbit.eu/17082007-what-are-elf-libraries-good-for-anyway/
https://testbit.eu/17082007-what-are-elf-libraries-good-for-anyway/#commentsFri, 17 Aug 2007 20:11:22 +0000http://blogs.gnome.org/timj/2007/08/17/17082007-what-are-elf-libraries-good-for-anyway/[...]]]>Or: Who in the world is actually linking against GLib without Gtk+?

I’m currently in the process of building some general purpose infrastructure for Rapicorn and because a good portion of the C++ utilities used in Beast and Rapicorn are factored out into an extra library called Birnet. I’m running into similar complexity issues here that we encountered with libglib vs. libgobject vs. libgdk already. I.e. lower level modules cannot reference or use mechanisms provided at a higher level (for instance deriving GHashTable from GObject). This becomes even worse with C++ which lacks support for partial classes (a class definition spanning multiple source files, like implementing gtk_widget_foo() outside of libgtk).

One remedy would be to actually merge libbirnet and librapicorn into a single project and let Beast depend on librapicorn. However, that’s essentially like we started out with GLib and Gtk+ in late 1996 when it got split out of the Gimp package. In June 1998 we split off libglib into a separate package for programs needing C utilities but not libgtk. So now, I’m reconsidering this separation after almost a decade. Reflecting on the move is probably not a bad idea before attempting the opposite with Rapicorn:

How many programs are actually linking against libglib but not libgdk, and is there any real benefit from separation? If your machine has libglib and a few spare minutes, you can use this command line to find out on your own:

That is, the servers have a quite limited set of GUI applications installed, but across a full fledged desktop the vast majority of applications is linked against libgdk anyway. I found Stefan Westerfelds Amd64 KDE desktop particularly interesting: It actually has the most applications linking against GLib, prolly because it has Mono and libQtCore.so installed which both link against GLib these days. Does anyone actually have a system with libglib installed but not libgdk?

GtkGdkPangoAtkCairoGlib / GLib ratio: (4394512 + 857576) / 857576 = 6.12. So the “savings” turn out to be splitting off 5/6th of the GUI stack size. At best, a GLib-only program is saving 4.2MB that way. But then again, those are likely to be in memory already anyway. And most definitely, they reside in a library, available as const .text on the harddisk. To put that into perspective: We’re talking about only a handful megabytes here. In physical size actually less than a hires desktop background image, smaller than some icon themes, or roughly 10% of a full linux-2.6.8 kernel + modules binary footprint (41MB). Also symbol resolution performance doesn’t necessarily improve through splits, Ulrich Drepper has a few words on multiple DSOs during deployment.

Given that selecting Gtk+ subcomponents to allow selective shrinkage for size constrained embedded devices via configure.in options is on our TODO list anyway; for this set of libraries: Size is not worth a split!

Of course, other aspects not reflected in library size weigh in much more strongly. Such as the ability to build different user communities around projects (e.g. for Cairo vs. Glib) or the reduced dependency chain (sshfs and Qt wouldn’t depend on GLib if it drew in Pango or Gtk+). So while for the vast majority of applications the GLib split off doesn’t make a huge difference technically, software evolution arguments make all the difference, definitely justifying a separation. The GLib internal splits are definitely not worth the hassle and conceptual boundaries though, if we ever break binary compatibility again, merging the GLib internal libraries and respective pkg-config packages would be a good idea.

Back to my original problem… I’m now pretty convinced that merging Rapicorn with the Birnet utility library will most likely not pose a problem anytime soon. And Qt 4 kindly demonstrates that splits can also be carried out successfully later on during the game.

]]>https://testbit.eu/17082007-what-are-elf-libraries-good-for-anyway/feed/717.07.2007 OpenGL for Gdk/Gtk+https://testbit.eu/17072007-opengl-for-gdkgtk/
https://testbit.eu/17072007-opengl-for-gdkgtk/#commentsTue, 17 Jul 2007 19:22:33 +0000http://blogs.gnome.org/timj/2007/07/17/17072007-opengl-for-gdkgtk/[...]]]>The idea of using OpenGL as a future core rendering architecture for Gtk+ has been brought up a couple times at GUADEC (and then some variations thereof). However there are good reasons to avoid that and major issues with the suggested approaches, in particular these need to be considered:

1) A dependency on OpenGL for a library as portable and as widely used as Gtk+ these days, could only ever be a weak one. That is, OpenGL features may or may not be used, depending on whether the current platform a Gtk+ application runs on actually has OpenGL available or not (e.g. by animating widgets conditionally to only be carried out when acceleration support is present).

2) The OpenGL 2D drawing API is effectively unusable for Gdk/Gtk+ drawing primitives. The main problem here is that OpenGL doesn’t provide pixel-perfect 2D drawing operations which are necessary for accurate input event processing and a coherent visual presentation (it also doesn’t always provide anti-aliasing in the 2D drawing API either). Here is a very good web page with nice screenshots summarizing the problems with OpenGL pixel-perfectness: OpenGL: “not pixel exact”, Hardware AntiAliasing.

3) By using XRENDER and hardware-accelerated X drivers, Cairo is already being performance optimized to utilize hardware acceleration. Trying to use a portable OpenGL subset instead (pixel shaders / triangle rendering) would be fairly pointless, it’d effectively be using the available portable hardware acceleration facilities through another indirection. So with more and more Gtk+-based platforms/applications moving to Cairo-based drawing, there is no additional infrastructure or support code needed to make use of available hardware acceleration facilities. Essentially, the portably usable hardware acceleration subset is brought to you automatically through Cairo and X.

]]>https://testbit.eu/17072007-opengl-for-gdkgtk/feed/1013.07.2007 Switch On Strings In C And C++https://testbit.eu/13072007-switch-on-strings-in-c-and-c/
https://testbit.eu/13072007-switch-on-strings-in-c-and-c/#commentsSat, 14 Jul 2007 10:46:38 +0000http://blogs.gnome.org/timj/2007/07/14/13072007-switch-on-strings-in-c-and-c/[...]]]>For Rapicorn i need a simple and nicely readable way to special case code on various strings, which isn’t a problem in most scripting languages but is not provided by C/C++. Switching on strings turns out to be a widely investigated topic: SOS (agay), TheScript, JavaBug (CNFW), CodeGuru, CLC-FAQ and google has lots more.

Now here goes my approach to switching on strings for C/C++, caveats upfront:
– The strings may only consist of identifier chars: [A-Za-z_0-9];
– Convenient application is based on a GNU Preprocessor feature: -imacros;
– Macro implementations use a GCC-ism: Statement Expressions;
– The convenient command line variant uses a bash-ism: Process Substitution;
– One compile-time script is needed, currently implemented in python (though it could be written in any language).

With this out of the way, let’s dive into the code. This is the syntax:

This is how the code needs to be compiled: gcc -Wall -O2 -imacros <(sosid.py <input.c) input.c

The actual implementation is fairly simple, sosid.py extracts all identifiers from SOSID() statements and generates a handful of macro definitions: SOSID(), SOSID_LOOKUP(), SOSID_LOOKUP_ICASE(), ... Plus input specific macros: SOSID__hello, SOSID__world, ... GCC will pick up those definitions via the -imacros option. That is enough to implement SOSID_LOOKUP() so that it can yield the integer IDs that the SOSID(*)/SOSID__* statements expand to for any string that corresponds to one of the SOSID(identifiers) occurring in the source file.

The lookups are implemented via a binary search and the strings are sorted in a way so that binary searches for case sensitive and case insensitive lookups are both possible. That means for large string lists, the lookups which are of complexity O(ld N) actually have a performance advantage over lengthy if..else if..else constructs with O(N) complexity.
The code is available as a single script at the moment: sosid.py.

Note that the bash-ism, imacros-use and GCC-isms are not mandatory, ordinary temporary files can be generated by the script and be included instead and it could be adapted to generate portable inline functions. The identifier-chars restriction is a hard one though. It comes from the fact that SOSID(ident) must yield valid C/C++ integers, and that can only be accomplished by token pasting. Depending on GCC is good enough for Rapicorn, so people will have to express active interest in this, in order for me to make the script more portable.

]]>https://testbit.eu/13072007-switch-on-strings-in-c-and-c/feed/729.06.2007 YummiYummiGithttps://testbit.eu/29062007-yummiyummigit/
https://testbit.eu/29062007-yummiyummigit/#commentsSat, 30 Jun 2007 02:51:22 +0000http://blogs.gnome.org/timj/2007/06/30/29062007-yummiyummigit/[...]]]>Starting with the discontinuation of cogito, i began to wonder about my future git usage patterns. On the one hand, i have become quite used to the convenience of some of the cogito commands, on the other hand i found myself using or looking up git commands every other day. Since the discontinuation, cogito cannot be expected to remedy those work-flow interruptions any time soon, so i actually started to cook up my own small set of convenience wrappers. I’m adding those on demand whenever i need a new source repository command, and i try hard to keep it simple, avoid giving the wrappers any options and keep an eye on them being very shell-completion friendly.

For the curious, the prefix ‘yy’ was chosen to allow conflict free shell completion. I’ll quote the yyhelp information here, so people can decide if it’s interesting for them and also get to see the installation instructions. I’m not entirely sure where to take this shallow wrapper in the future, however keeping it simple is really the main incentive behind it. Any deeper logic required should rather be filed as git-core requests. If you want to talk to me about YummiYummiGit, drop me a line via email, or try #git on irc.freenode.net:6667.

]]>https://testbit.eu/28062007-searchable-api-docs/feed/113.06.2007 Back Onlinehttps://testbit.eu/13062007-back-online/
https://testbit.eu/13062007-back-online/#commentsWed, 13 Jun 2007 00:55:16 +0000http://blogs.gnome.org/timj/2007/06/13/13062007-back-online/[...]]]>I just got back online after a full week of DSL-outage due to the Deutsche Telekom Strike. If you are awaiting response via email or bugzilla from me, please be patient, I’m doing my best to catch up. Feel free to poke me about urgent issues via IRC/IM though, or drop me a reminder if you haven’t heard back from me in another week.

]]>https://testbit.eu/13062007-back-online/feed/104.06.2007 LinuxTaghttps://testbit.eu/04062007-linuxtag/
https://testbit.eu/04062007-linuxtag/#commentsMon, 04 Jun 2007 21:28:42 +0000http://blogs.gnome.org/timj/2007/06/04/04062007-linuxtag/[...]]]>As announced by Hallski, Sven, Mitch and me went to LinuxTag 2007 and operated the Imendio and GNOME booths. As usual, Sven did a great job nurturing random users approaching the booth, together with Mathias Hasselmann and Michael Köchling. I put more focus on the amazingly large program of the conference, of which I’ll give a short roundup here.

The Xen keynote was on Wednesday. Xen 3.0 comes with some interesting new features, it’ll introduce IO virtualization and supports new virtuallization technologies from Intel and AMD. One important lesson I took away from that session is that using virtualization aware drivers on the guest OS can boost performance from roughly 10% using generic virtualization techniques, to more than 90% of the ideal performance throughput (native host OS performance).
–
This day, there was also the ZODB3 talk. This is an object database which can be used completely independently from Zope, it provides a very nice interface in python to implement hierarchical object tree persistence, has ACID transactions with rollbacks and allows for doomed transactions. At the lower level it uses a stable subset of python’s pickle and supports multiple storage backends.

Thursday had a Linux forum where Thomas Gleixner discussed recent realtime work in the kernel. The low latency and preemption patches that went into the kernel over the 2.6.x series brought a number of positive side effects such as general responsiveness improvements on loaded systems and new debugging mechanisms. From the new debugging facilities and raised threading issue awareness amongst kernel developers, a good 1200 patches containing bug fixes and cleanups resulted. And development in this are has not come to a halt, current/future work includes getting rid of idle-state interrupts that do nothing by having a tick-less kernel that only wakes up every once in a while when actual work is due and cleaning up general jiffie dependent code in the kernel. Now what’s left to hope is that distributors get their act together and enable the low-latency preemption patches for their desktop kernels. The patches work, they are stable, and they provide a much better user experience. E.g. if you experience sluggish system behavior during crypto filesystem IO, or experience drop outs with your sound system/server, don’t blame it on the kernel people, blame it on your distribution not allowing time slice preemption.
–
The other talk I attended was held by Matthias Hopf about the Compiz+Beryl merger. The resulting effects he presented excited the crowd as usual, and then he talked some about ongoing developments like input event transformation. After the talk we had some more personal chatter about using the 32bit XRGB visual to add alpha channels to XWindows, and future X extensions to allow applications to notify the server about when to issue EXT_texture_from_pixmap (needed for flicker free composite support).

On Friday, Lennart gave a very good presentation on the state of PulseAudio. He described how it solves the vast majority of audio use cases and can in combination with libsydney finally put an end to the never-ending lack of a portable and usable audio API. Beyond the talk, Lennart and I used every spare minute during the LinuxTag days to discuss libsydneys new API. All in all, it looks like a suitable candidate to replace (or continue) the effort Stefan and I started with CSL (we later suspended the project assuming PortAudio would fulfil the role) to make sound backends transparent.
–
After lunch, Quim Gil presented present and future of the Maemo platform. The points I personally found most notable are:
* libhildon2 is going to become an upstream community project, using Gnome infrastructure like bugzilla, with Nokia providing the core developers.
* Future platform updates (applications and OS) should be possible via APT, so flashing becomes a secondary upgrade method.
* Nokia is currently collecting feature requests for the Maemo platform. They’ll be integrated into Nokias platform plans where possible, so if you have any input to provide, state it here: Maemo Roadmap Page.
* The Maemo versioning scheme now uses alphabetical letters to indicate versioning progress. The current/upcoming versions are:
– [B]ora: current platform (3.x);
– [C]hinook: next platform (4.x) based on Gtk+-2.10, comes with SDKs before launch;
– [D]iabolo: intended to keep API/ABI from here on, unless upstream also breaks;
– [E]lephanta: SBox2 might be available at this time.
* Nokia tries to open up as much of the Maemo platform as possible and they will try to reduce dependencies between opened software and closed platform components in the future. The reason that some programs stay closed anyway are: a) missing legal clearance or licensing for some code portions – opening up code in one division of the company might affect legal claims (IP) on closed components developed and supported in another division); b) code may be subject to hardware vendor NDAs.
* Nokia will embrace attempts such as getting Maemo to run on different hardware in the future, or running non-Maemo software on the N770/N800.
* Nokia intends to introduce abstraction layers for hardware specifics (N770 vs. N800 vs. future devices) where possible.
–
The closing talk for this day was the Gimp presentation by Simon Budig, titled “Pimp My Gimp”. Simon presenting new Gimp features has become somewhat of a LinuxTag tradition over the years, and as usual new Gimp features managed to excite the crowd, even though we also had the usual glitches like a perfectly tested new clone-tool refusing to work on stage.

Saturday morning started with a vivid presentation on DBus by Marcel Holtmann. Though no stunning new features were presented, he did a good job on introducing the overall architecture and getting the audience
hooked up for DBus applications with his presentation and some simple example code.
–
After noon, there was a big podium discussion on preventive data mining of personalized records, currently planned to be realized in upcoming laws by the German government. While the forum was quite interesting and definitely necessary to have at a conference like this, the discussion didn’t present surprisingly new findings for anyone following matters already. The discussion was great however, mostly because of the strong involvement of a rather large audience. It was pointed out that there is a massive lack of public awareness for the incredible data mining hunger of the government organizations and certain big companies, and that this is one of the topics that are rather uneasy to educate the majority of the democratic republic on.
–
Later during the day, Jono Bacon gave his obligatory Ubuntu talk about how Ubuntu cares about user experience and about growing the community. Personally, I still feel that more involvement from Canonical in genuine upstream development such as e.g. the Gtk+ project would be better in the long run for both Canonical and the upstream eco systems.
–
Finally, there was the Open Source Press talk which started out on how CopyLeft used to be the natural state for information exchange since the beginning of mankind, until publishers came into existence in the 16th century and invented CopyRight restrictions to preserve their investments in terms of hardware and human resources. It then was suggested that in a completely networked future computer age, we’ll be back to a pure CopyLeft situation. The presenter didn’t really seem to apply his findings though, when in the second half of the session he assumed that all mass-digestible texts must be edited by publishers and thus unconditionally require copyright laws. He also didn’t want to acknowledge that books existed which are sold and published online or that large amounts of interesting text (>= 500 pages) could possibly be written by non-publishers. While this presentation showed that publishers do recognize and ponder about open content these days, it also made clear that they still have a lot to learn.

As a closing note, I’d like to say a big Thank You to the LinuxTag Orga-Team. All throughout the four days there was an exceptionally good program in five presentation rooms, with up to four additional forum presentations distributed across the booth areas. From our perspective overall event organization went pretty well, this also resonated in German media.

]]>https://testbit.eu/04062007-linuxtag/feed/225.04.2007 Call For Helphttps://testbit.eu/25042007-call-for-help/
https://testbit.eu/25042007-call-for-help/#commentsWed, 25 Apr 2007 00:09:36 +0000http://blogs.gnome.org/timj/2007/04/25/25042007-call-for-help/[...]]]>Gtk+ Patch Testing & Committing: Recently i was caught on IRC and asked to look into a few outstanding Gtk+ bug reports. Some of them easy, some of them not so easy, e.g. one required a lengthy in-depth evaluation of interacting code while some of the others already had patches applied. That particular moment, I barely had any spare time to look into Gtk+ bugs and was actually on my way to the shower and then head downtown. We still managed to squeeze in and close the bunch of 3 or 4 bug reports in less than an hour however, because people actually helped me in applying, compiling, testing and committing the changes. While core maintainer approval was a necessity for some resolutions, there was much more work to be put into each report beyond giving the OK to the final patch versions. In particular the patch applying, adaptions to recent trunk, compiling it, running the test suite or test cases and committing is something that many developers who are not overly familiar with the code base can still do. This is a great way to learn and get into a project, and to help out an understaffed core team. It is a great way to contribute to your pet project and will help to make your visions and ideas about it come true.

Gtk+ has a volunteer task list up on the wiki. One of the open positions is the one i just described, and it’d much help the project if people signed up for it. Note that having SVN access can help but is not a requirement for all tasks, access rights can be granted for active contributors.

Next time you wonder why your Gtk+ bug report seems stalled, don’t ask why it’s being processed so slowly, ask yourself why you’re not helping the project already, so all submissions can be processed faster, including yours.

In other news, there’s been some discussion on width-for-height and height-for-width management around Gtk+ recently. So I summarized the approach taken by Rapicorn, which in a nutshell allows arbitrary constrains with multiple resizing iterations and employs dedicated strategies to stabilize the resizing process: Hysteresis problems in Gtk+ – Rapicorn size negotiation.

This should help to clear up some of the motivations behind the Rapicorn project, in particular how it relates to Gtk+ and why it aims at implementing features that are to some extend already covered by Gtk+ or other projects like GnomeCanvas.
Here is a small excerpt to wetten the appetite:

These days Gtk+ is [...] a very successful toolkit.
However, maintenance and continuous development of a project at this scope
and scale come at significant costs in terms of development flexibility
and compatibility. It is not anymore a suitable place for evolution of new
experimental GUI technologies and quick paradigm shifts. So radically new
toolkit approaches or design ideas now need to be explored elsewhere and
only if successful and applicable can reflect back onto Gtk+ development.
-
Rapicorn explores some approaches which are simply different from
established Gtk+ paradigms.

]]>https://testbit.eu/05042007-rapicorn-website/feed/126.03.2006 Beasty Bitshttps://testbit.eu/26032006-beasty-bits/
https://testbit.eu/26032006-beasty-bits/#respondSun, 25 Mar 2007 19:17:54 +0000http://blogs.gnome.org/timj/2007/03/25/26032006-beasty-bits/[...]]]>I will quickly roll up some of the interesting bits that happened around Beast in the last couple of weeks. Stefan Westerfeld sat down and put up a collection of the various instruments and loops he is producing with Beast. Nicely described BSE files along with Ogg previews are available in his music collection: STW Music Archive.

]]>https://testbit.eu/26032006-beasty-bits/feed/022.01.2007 Rapicorn-0.1.2https://testbit.eu/22012007-rapicorn-012/
https://testbit.eu/22012007-rapicorn-012/#respondMon, 22 Jan 2007 14:42:04 +0000http://blogs.gnome.org/timj/2007/01/22/22012007-rapicorn-012/[...]]]>The last Rapicorn snapshot had a couple build and compiler issues which could be fixed meanwhile, so here is the “real” 0.1.2 release: rapicorn-0.1.2.tar.gz It’s still a technology preview release with known issues, nevertheless useful to look at. The release NEWS:

]]>https://testbit.eu/22012007-rapicorn-012/feed/028.12.2006 G_SLICE=debug-blockshttps://testbit.eu/28122006-g_slicedebug-blocks/
https://testbit.eu/28122006-g_slicedebug-blocks/#commentsTue, 02 Jan 2007 09:08:32 +0000http://blogs.gnome.org/timj/2007/01/02/28122006-g_slicedebug-blocks/[...]]]>One of the typical problems with implementing an allocator is that all sorts of memory failures in programs get attributed to the allocator. That’s because messing up heap memory somewhere in a program or library usually messes up the allocator state even if it is implemented correctly. For several months now, we have been trying to track down and fix a very nasty memory corruption issue in Beast (#340437), which could be hunted down to rare and racy triggering with GSlice enabled, but not with G_SLICE=always-malloc set. Valgrind’s memchecker didn’t help with this particular Beast bug and couldn’t possibly track all invalid GSlice uses.

So i sat down to implement a slice address and size validator to catch the most frequent GSlice misuses. The debugging validator consists of a flat fixed-size hashing tree with fairly big prime-sized nodes and binary searchable arrays to manage collisions in the hash buckets. This structure avoids circular dependencies with GSlice (like GHashTable and GTree would have it) and is still reasonably easy to implement. The hashing makes use of the approximately random distribution of allocator block addresses in the lower bits but fairly sequential distribution in the higher bits, so the structures are reasonably space efficient and can adapt in size incrementally with every new megabyte handed out by malloc(3)/memalign(3). This also keeps hash collisions on an acceptably low average for allocations within a 4GB address range. Using a single global structure keeps GSlice from scaling well across multiple threads, but i think that’s fair enough for a debugging mode validator. Quickly after running an early version of this validator, the cause of the above Beast bug was found and could be fixed.

Another issue that some people might run into with the recent GSlice code is:

***MEMORY-WARNING***: GSlice: g_thread_init() must be called before all
other GLib functions; memory corruption due to late invocation of
g_thread_init() has been detected; this program is likely to crash,
leak or unexpectedly abort soon...

]]>https://testbit.eu/28122006-g_slicedebug-blocks/feed/127.12.2006 BEAST v0.7.1 out of the doorhttps://testbit.eu/27122006-beast-v071-out-of-the-door/
https://testbit.eu/27122006-beast-v071-out-of-the-door/#respondWed, 27 Dec 2006 20:34:46 +0000http://blogs.gnome.org/timj/2006/12/27/27122006-beast-v071-out-of-the-door/[...]]]>It really has been a long way to get this release out of the door. It fixes some serious crashers that i intend to blog about later, but more importantly it fixes a security vulnerability issue, here’s the related advisory: artswrapper vulnerability 2006-2916 Upgrading from any older Beast version is thusly strongly recommended. I’ll not get into too many boring details here, so let me just say that Beast now supports different musical tuning systems and also ships with new and extended modules (BseQuantizer). All the g(l)ory details can be found in the original announcement: ANNOUNCE: BEAST/BSE v0.7.1

]]>https://testbit.eu/27122006-beast-v071-out-of-the-door/feed/020.12.2006 Gtk+ Core Maintainer Shortagehttps://testbit.eu/20122006-gtk-core-maintainer-shortage/
https://testbit.eu/20122006-gtk-core-maintainer-shortage/#commentsWed, 20 Dec 2006 21:49:01 +0000http://blogs.gnome.org/timj/2006/12/20/20122006-gtk-core-maintainer-shortage/[...]]]>Recently there have been complaints about a lack of quality assurance in the Gtk+ project and introduction of regressions with releases. Also the pace at which bugs and feature requests are being addressed may not be to everyones liking. The reasons for these issues are manifold, some are persisting for a long time and aggravated, others emerged only recently. Some stem from the code structure, others from the variety of interests different parties have in Gtk+, and the list goes on… I think in a nutshell, it can all be boiled down to a single concise fact:

Gtk+ has an alarming shortage on core maintainer resources.

That being said, let me make one thing crystal clear here: this can not be blamed on the core maintainers themselves, they are doing their best to cope with the situation as it is. By a large ratio, there simply isn’t enough man power available to handle all the incoming requests, to review all contributed patches, debug all the bugs filed and to do important nurturing and maintenance work needed by an ever growing code base (like refactoring). This situation has been persisting for a longer time now, and has led to significant aging in the code base and affected the overall quality of the project. As time passes, it’ll just become worse if we’re not going to put more resources at the project as a community. I figured that in order for this to happen, a sense for the problematic nature of the current maintenance situation has to first be developed in the community, which is why i posted to gtk-devel and related lists yesterday:

]]>https://testbit.eu/20122006-gtk-core-maintainer-shortage/feed/129.11.2006 Rapicorn snapshothttps://testbit.eu/29112006-rapicorn-snapshot/
https://testbit.eu/29112006-rapicorn-snapshot/#commentsTue, 28 Nov 2006 19:23:06 +0000http://blogs.gnome.org/timj/2006/11/28/29112006-rapicorn-snapshot/[...]]]>So Herzi pressed me to make another Rapicorn snapshot available. The result of which is now up here: rapicorn-0.1.2-snapshot.tar.gz It’s still very much a technology preview, but shaping well in various aspects. Here’s the release NEWS:

Known Issues: + instabilities upon main loop exit. + possible build issues with g++ versions other than 3.4.

]]>https://testbit.eu/29112006-rapicorn-snapshot/feed/324.11.2006 How to start a web browserhttps://testbit.eu/24112006-how-to-start-a-web-browser/
https://testbit.eu/24112006-how-to-start-a-web-browser/#commentsFri, 24 Nov 2006 16:30:28 +0000http://blogs.gnome.org/timj/2006/11/24/24112006-how-to-start-a-web-browser/[...]]]>Last week i got a bug report where activating “Help/Quick Start”, “Help/FAQ” and any other web browser related menu item in Beast would do nothing. That of course completely defeats the purpose of writing help documents and offering help via the menu. Luckily, we were able to get browser launching debugging print outs in that scenario, and what happened was: * Beast started up gnome-open with the help URL. * Since gnome-open was present on the system and could be started, browser launching was considered successful and no error dialog had to be displayed. * gnome-open in turn would error out (for unknown reasons):

Error showing URL: There was an error launching the default action command associated with this location.

The problem here is clearly that Beast wrongly assumed launching a browser or script successfully would be enough. I’ve since reworked the logic so that: * Browser launch scripts/programs are now always started in the foreground to verify their respective exit codes. * A correctly functioning browser launcher now has to reliably start the browser in the background and has to provide a correct exit code. * Graphical browser programs are started in the background, because browsers can reasonably be expected to show a URL once started successfully. Besides that, the exit code depends on other events during a browsing session anyway and it also is only provided at the end of a browsing session. * If no browser launching script with 0 exit code could be started and no browser program could be started successfully in the background, a descriptive error dialog is shown, including the target URL.

I had to test out a bunch of browser launchers to get this to work reliably, and some of the launchers that fell short to be usable really came at a surprise: * gnome-open (GNOME) – the browser is opened in the background, and if no browser could successfully be launched a non-0 exit code is provided. * exo-open (XFCE) – browser is opened in the background, provides correct exit code. * kfmclient (KDE) – browser is opened in the background, provides correct exit code. * gnome-moz-remote (old GNOME) – browser is opened in the background, provides correct exit code. * browser-config (Gentoo) – this script goes through some lengths to make sure all browsers are opened in the background, but then it always quits with an exit code of 0, even if no browser was configured and launched. * xdg-open (Portland) – provides correct exit code, but the browsers are unpredictably launched in the foreground or background (usually depending on whether a running browser instance already exists or not). * sensible-browser (Debian) – provides correct exit code, but unpredictably opens in foreground or background. * htmlview (Red Hat) – provides correct exit code, but unpredictably opens in foreground or background.

In the end, the Linux distribution launchers that i was really hoping for to get user configuration right (honor $BROWSER) and work reliably across desktops failed short to do so (browser-config, xdg-open, sensible-browser and htmlview), while all the desktop project launchers fullfiled the required needs (gnome-open, exo-open and kfmclient). For everyone interested, the current browser launching code used in Beast now can be found here: Beast: birnet/birnetutils.cc – scroll to “url_test_show“.

]]>https://testbit.eu/24112006-how-to-start-a-web-browser/feed/123.10.2006 Beast and unit testinghttps://testbit.eu/23102006-beast-and-unit-testing/
https://testbit.eu/23102006-beast-and-unit-testing/#respondMon, 23 Oct 2006 12:38:52 +0000http://blogs.gnome.org/timj/2006/10/23/23102006-beast-and-unit-testing/[...]]]>There’s been quite some hacking going on in the Beast tree recently. Stefan Westerfeld kindly wrote up a new development summary which is published on the Beast front page.

In particular, we’ve been hacking on the unit tests and tried to get make check invocations run much faster. To paraphrase Michael C. Feathers from his very interesting book Working Effectively with Legacy Code on unit tests:

Unit tests should run fast – a test taking 1/10th of a second is a slow unit test.

Most tests we had executed during make check took much longer. Beast has some pretty sophisticated test features nowadays, i.e. it can render BSE files to WAV files offline (in a test harness), extract certain audio features from the WAV files and compare those against saved feature sets. In other places, we’re using tests that loop through all possible input/output values of a function in brute force manner and assert correctness over the full value range. Adding up to that, we have performance tests that may repeatedly call the same functions (often thousands or millions of times) in order to measure their performance and print out measurements.

These kind of tests are nice to have for broad correctness testing, especially around release time. However we did run into the problem of make check being less likely executed before commits, because running the tests would be too slow to bother with. That of course somewhat defeats the purpose of having a test harness. Another problem that we ran into were the intermixing of correctness/accuracy tests with performance benchmarks. These often sit in the same test program or even the same function and are hard to spot that way in the full output of a check run.

To solve the outlined problems, we changed the Beast tests as follows:

* All makefiles support the (recursive) rules: check, slowcheck, perf, report (this is easily implemented by including a common makefile).

* Tests added to TESTS are run as part of check (automake standard).

* Tests added to SLOWTESTS are run as part of slowcheck with --test-slow.

* Tests added to PERFTESTS are run as part of perf with --test-perf.

* make report runs all of check, slowcheck and perf and captures the output into a file report.out.

* We use special test initialization functions (e.g. sfi_init_test(argc,argv)) which do argument parsing to handle --test-slow and --test-perf.

* Performance measurements are always reported by the treport_maximized(perf_testname,amount,unit) function or the treport_minimized() variant thereof, depending on whether the measured quantity is desired to be maximized or minimized. These functions are defined in birnettests.h and print out quantities with a magic prefix that allows grepping for performance results.

* make distcheck enforces a successful run of make report.

Together, these changes have allowed us to easily tweak our tests to have faster test loops (if !test_slow) and to conditionalize lengthy performance loops (if test_perf). So make check is pleasingly fast now, while make slowcheck still runs all the brute force and lengthy tests we’ve come up with. Performance results are now available at the tip of:

The results are tailored to be parsable by performance statistics scripts. So writing scripts to present performance report differences and to compare performance reports between releases is now on the TODO list.

]]>https://testbit.eu/23102006-beast-and-unit-testing/feed/030.09.2006 Monster out of Closethttps://testbit.eu/30092006-monster-out-of-closet/
https://testbit.eu/30092006-monster-out-of-closet/#respondSat, 30 Sep 2006 09:28:23 +0000http://blogs.gnome.org/timj/2006/09/30/30092006-monster-out-of-closet/[...]]]>Rodney Dawes blogged in reply to my previous blog entry on Hammering the intltool trouble. First of all, blog discussions suck, they suck less if they are carried on in the comments section of a blog entry, but i agree with Rodney that this is still not as helpful as Bugzilla or email conversations are. Coincidentally, the actual issue, namely Beast‘s intltool patch has been addressed in a private email discussion in 2005 and 2006 in which Rodney got Cc:-ed and also replied. That thread also linked to a version of the Beast intltool-extract.in patch. It didn’t seem like this would go into intltool though, so we simply continued to maintain our patched version. In any case, if Rodney prefers to have this issue tracked by Bugzilla, that’s fine, here’s the new bug report which describes our scenario and has the patch attached: Bug 358495 – support for multiline scheme strings.

]]>https://testbit.eu/30092006-monster-out-of-closet/feed/028.09.2006 Hammering the intltool troublehttps://testbit.eu/28092006-hammering-the-intltool-trouble/
https://testbit.eu/28092006-hammering-the-intltool-trouble/#respondThu, 28 Sep 2006 15:35:57 +0000http://blogs.gnome.org/timj/2006/09/28/28092006-hammering-the-intltool-trouble/[...]]]>So I’ve finally had enough trouble with kludging around the different bits and odds of intltool. For several years now, the Beast project has to maintain a patch against intltool to apply after intltoolize --force --copy. The patch must be applied in order to enable translation of multi line scheme strings, and to allow custom recursive Makefile rules (which usually break due to po/Makefile.in.in). Most often the patches would not apply on systems other than mine, because they didn’t cover all stock intltool versions or would still fail for matching versions because of conflicts with patches applied to intltool during distribution packaging. The latest annoyance we ran into were m4 macro incompatibilities that for a variety of reasons were once more an issue not easily worked around.Beast SVN now simply contains the copied and patched intltool files (0.35.0) so other developers are finally able again to build Beast from SVN without any autogen.sh trouble and pass all of make check (which also checks whether scheme strings got properly merged into .po files). After all, it’s not that many files anyway: acintltool.m4 intltool-extract.in intltool-merge.in intltool-update.in po/Makefile.intltool

]]>https://testbit.eu/28092006-hammering-the-intltool-trouble/feed/030.08.2006 Scroll-Area and Slidershttps://testbit.eu/30082006-scroll-area-and-sliders/
https://testbit.eu/30082006-scroll-area-and-sliders/#respondWed, 30 Aug 2006 18:44:12 +0000http://blogs.gnome.org/timj/2006/08/30/30082006-scroll-area-and-sliders/[...]]]>It has been a while since i last blogged about Rapicorn and worked on new features for it. So here are some updates.

In order to get a scrollable area implemented, i first started out with the h/v slider implementations. That in turn required sorting out of the basic threading and main loop models (Rapicorn supports thread-per-window mode) to get timers going. I then ended up implementing 3 different gadgets to make up the sliders: SliderArea, SliderTrough and SliderSkid. I managed to keep the code generic enough with regards to packing abilities and movement to support horizontal and vertical scales as well as scrollbars with these 3 gadgets.

For the curious, an XML excerpt that packs together the necessary frames, buttons, boxes, ambience (shading) and functional gadgets for a vslider is here: Rapicorn vslider definition (23 lines).

Since Rapicorn currently has to support Gtk+/Gdk >= 2.6.x as rendering backend, scrolling had to be implemented with gdk_draw_drawable() + gdk_event_get_graphics_expose(). Once this dependency can be raised, we’ll use gdk_window_move_region() however. Tonight, i managed to complete the coordinate offset handling and got event processing going inside the scroll area, so here’s a button toggled inside a scroll area:

A couple other things also got fixed, like the Appearance/ColorScheme handling. E.g. here’s a blue test window, where the color scheme is derived from a set of just 3 base colors:

I’ll probably put up another tarball soon, after some polishing of the build setup and the window/main loop destruction phases.

]]>https://testbit.eu/30082006-scroll-area-and-sliders/feed/016.07.2006 BEAST/BSE v0.7.0 finally outhttps://testbit.eu/16072006-beastbse-v070-finally-out/
https://testbit.eu/16072006-beastbse-v070-finally-out/#respondSun, 16 Jul 2006 18:05:39 +0000http://blogs.gnome.org/timj/2006/07/16/16072006-beastbse-v070-finally-out/[...]]]>After next Friday, Stefan Westerfeld and me are leaving for a 3 week vacation. So in order to not let the Beast release slip even more, we’ve been working hard over the last couple weeks to get a new tarball out of the door. After much fiddling and an SVN-migration hick up in the last minute (Beast is in SVN now and will stay there), it’s finally accomplished:

We really hope people are going to have fun with this release and report all bugs they encounter in Bugzilla.

]]>https://testbit.eu/16072006-beastbse-v070-finally-out/feed/021.06.2006 Encrypting the Laptop for GUADEChttps://testbit.eu/21062006-encrypting-the-laptop-for-guadec/
https://testbit.eu/21062006-encrypting-the-laptop-for-guadec/#respondWed, 21 Jun 2006 19:14:58 +0000http://blogs.gnome.org/timj/2006/06/21/21062006-encrypting-the-laptop-for-guadec/[...]]]>Traveling with a laptop to a foreign country, waiting at airports, driving in cabs, etc. puts your data at certain risks, considering the number of laptop losses at public places. Now, most of the documents and code on my system is free software or not particularly private anyway, but unfortunately not all. Some work related documents and emails i’m carrying around are not allowed to be disclosed, and so i had to decide whether to identify each one of them and either encrypt or delete them individually (very unattractive to maintain for emails and an automatically sync-ed homedir), or encrypt the laptop harddisk as a whole.
I went for the latter, luckily newer Linux kernels (>= 2.6.4) are meant to provide the necessary mechanisms out of the box. And except for one nasty devfs issue, migrating my Debian sarge system to fully encrypted partitions went surprisingly well. It still took some time to figure the necessary bits involved, especially since initrd-tools requires devfs support which got removed from newer Linux kernels. So i’ve kept notes on the steps performed and put them up here: Debian Rootfs Encryption Notes, mkinitrd 1.201 nodevfs hack.

]]>https://testbit.eu/21062006-encrypting-the-laptop-for-guadec/feed/021.05.2006 LAC Proceedingshttps://testbit.eu/21052006-lac-proceedings/
https://testbit.eu/21052006-lac-proceedings/#respondSun, 21 May 2006 19:16:25 +0000http://blogs.gnome.org/timj/2006/05/21/21052006-lac-proceedings/[...]]]>The LAC 2006 Proceedings are now online: LAC 2006 Proceedings as PDF
I especially want to recommend the Realtime Audio vs. Linux 2.6 paper by Lee Revell to everyone who has an interest in low latency audio processing on Linux. At the conference, Lee gave an excellent presentation on the topic and the recent work that went on in the 2.6 series of the Linux kernels. He also presents curiosities like why benchmark optimized video card drivers may get into the way of realtime audio processing.
Amongst other things at the Linux Audio Conference i recently blogged about, i also found the presentation on Ontological Processing of Sound Resources particularly interesting. With the growth of audio material collections, searches confined to the natural hierarchy enforced by file systems become increasingly ineffective. An alternative browsing approach can be based on tagging of audio material (the paper examines this in the context of instrument types), automatic deduction of tags and inference mechanisms on these tags. This is suitable to support user queries that contain music/instrument characteristics rather than arbitrary names.

In other news, Stefan Westerfeld summed up our recent development activities in the BEAST News, which may be suitable to sustain everyone eagerly awaiting the upcoming 0.7.0 release

]]>https://testbit.eu/21052006-lac-proceedings/feed/017.05.2006 The Beast Documentation Quest (3)https://testbit.eu/17052006-the-beast-documentation-quest-3/
https://testbit.eu/17052006-the-beast-documentation-quest-3/#commentsTue, 16 May 2006 19:23:48 +0000http://blogs.gnome.org/timj/2006/05/16/17052006-the-beast-documentation-quest-3/[...]]]>Faced with the documentation system requirements outlined in the last episode (The Beast Documentation Quest (2)) it took a while to figure that there really was no single documentation tool fulfilling all the needs outlined. So writing a new documentation tool for the Beast project looked like the only viable solution. I wasn’t ready to start out rolling a new full fledged C++ parser though… Fortunately, Doxygen includes an XML generation back-end, that will dump all the C/C++ structure after the parsing stage, including comments, into a set of XML files. Unfortunately it unconditionally preprocesses the source code comments. So it required some trickery to circumvent complaints about unknown or invalidly used Doxygen macros when combined with a newly developed macro processor… To get the macro processor going, i wrote a recursive descend parser for “@macro{arguments} more text\n“-style constructs in Python. Added up a syntax facility for new macro definitions, an algorithm to detect and generate sections plus a rudimentary HTML generation back-end, and voila – a new documentation generator was born: Doxer. That was around last summer.

In response to my last blog entry on this topic, one commenter wrote:

I think that starting another documentation project would be totally useless.

I don’t think i can agree here. For any software solvable problem you have, you can only write your own tool if you don’t find an existing one that covers your needs or could be extended to do so. Admittedly, i didn’t know about Synopsis back then, and i intend to investigate more of its innards at some point to figure whether it can be integrated with Doxer nicely. Until then, Beast does again have a working documentation tool, re-enabling us to keep documenting the source code and updating the website. At this point, the CVS version of Beast has been fully migrated to generate and display HTML runtime documentation, using a browser according to the users preferences. Also all source code documentation was converted and the website is fully generated from Doxer. All of this is driven by circa 6k lines of code, made possible only by the expressiveness of Python and by using Doxygen as a source code parsing backend.

]]>https://testbit.eu/17052006-the-beast-documentation-quest-3/feed/130.04.2006 Linux Audio Conferencehttps://testbit.eu/30042006-linux-audio-conference/
https://testbit.eu/30042006-linux-audio-conference/#respondSun, 30 Apr 2006 04:54:17 +0000http://blogs.gnome.org/timj/2006/04/30/30042006-linux-audio-conference/[...]]]>The 4th International Linux Audio Conference is quite fun, we got many interesting comments on our Beast Demo (here’s also a Beast Paper from the 2005 proceedings). One of the most requested feature seems to be Jack support for Beast. To quote Stefan on the conference: “Some talks were really cool, for example the Reverb stuff Fons Adriaensen is working on.” And he’s quite right: Fons wants to build up a free database of impulse responses of halls and rooms the community has access to, so reverb applications can offer a list of interesting room characteristics to choose from, and he’s providing free software and giving workshops to enable people to do those measurements.

]]>https://testbit.eu/30042006-linux-audio-conference/feed/018.04.2006 Fighting the build toolshttps://testbit.eu/18042006-fighting-the-build-tools/
https://testbit.eu/18042006-fighting-the-build-tools/#respondTue, 18 Apr 2006 15:27:35 +0000http://blogs.gnome.org/timj/2006/04/18/18042006-fighting-the-build-tools/[...]]]>A few months ago, i migrated the beast CVS module to automake-1.9. I’ve been working on fixing the various issues this change involved ever since…
At this point, I’m stuck in the end phase of getting make distcheck to work, and ran into a rather unpleasant issue:

What happens basically is: make distcheck in automake-1.9 creates a tarball, builds and installs the tarball, does make check and then uninstalls and checks no files are left after uninstall (which is a hairy issue in itself due to update-mime-database(1) leaving stale files). After that, it tries to also make dist distcleancheck and this is where the fun begins. In order to make dist in the build tree of the distributed tarball, po/Makefile.in attempts to update all .po files for which it needs $(GETTEXT_PACKAGE).pot (for beast, that’s beast-v0.7.0.pot). Creating the .pot file however does require all source files listed in POTFILES.in, and beast is deliberately NOT shipping all the source files listed in there.

Why is beast not shipping all POTFILES.in referenced sources?
The reasons for not shipping all source files in the main tarball are manifold, for the moment, let me just say we split up a beast release into multiple packages because different components are updated at different intervals, and because we need to partition dependencies reasonably. This is rather uninteresting for translators though who just need to translate the strings of “the beast project” as a whole. Which is why the beast CVS module has only one POTFILES.in that lists all translatable source files from CVS, regardless of the actual release tarball they are shipped in. This is obviously the best solution for the translators and for release maintenance.
But it is not a solution automake-1.9 likes.
At this point, I’m stuck at this problem being unsolved, I’m not even entirely sure who’s at fault here. Is the beast project really so wired in shipping multiple tarballs but wanting only a single translation entry point for translators? Or, is intltool at fault because it recreates beast-v0.7.0.pot instead of shipping it or leaving the .po files untouched? Or is it automake-1.9 which tries to ensure that build trees from distribution tarballs can pass make dist again?
I’m inclined to say giving up make dist for anything that is not a correctly built CVS tree would be the best for the next beast release and certainly weights in far less than giving up on any of the other preconditions that led to this problem. However I’m not so sure that removing make dist distcleancheck from automake’s distcheck target would make sense for other projects as well. And as things stand, automake-1.9 doesn’t allow for removing these targets from distcheck anyways…

Even though the whole situation is a bit involved, comments are very appreciated…

]]>https://testbit.eu/18042006-fighting-the-build-tools/feed/005.04.2006 Console Groupshttps://testbit.eu/05042006-console-groups/
https://testbit.eu/05042006-console-groups/#commentsWed, 05 Apr 2006 09:32:52 +0000http://blogs.gnome.org/timj/2006/04/05/05042006-console-groups/[...]]]>While i was reading up on the Anti-Grain Geometry project discussion of AGG vs. Cairo, i came across this link: The State of Linux Graphics.
This paper was apparently in the news last September, but somehow i managed to miss it. I’d say it’s a must read for anyone seriously involved with Linux graphics and even somewhat beyond that. Apart from disentangling the various rendering systems and approaches out there, it gives a good prospect of the technology needed for future desktops and hardware.
One particular thing that caught my attention is something that is needed in the Linux audio world as well: Console Groups.

Console groups are collections of hardware that go together to form a login console.

E.g. a mouse, keyboard, graphics display, sound card and maybe a specific USB port/hub and a printer.
This topic seems to have been raised at OLS 2005 although i wasn’t able to find something relevant in its proceedings. I’d appreciate any pointers readers may have on active work in this area.

]]>https://testbit.eu/05042006-console-groups/feed/123.02.2006 The Beast Documentation Quest (2)https://testbit.eu/23022006-the-beast-documentation-quest-2/
https://testbit.eu/23022006-the-beast-documentation-quest-2/#commentsFri, 24 Feb 2006 05:18:55 +0000http://blogs.gnome.org/timj/2006/02/24/23022006-the-beast-documentation-quest-2/[...]]]>In order to look for a new documentation tool for Beast, i set out a list of requirements that had to be matched:

a) We need to generate reference documentation from C and C++ source code comments. b) Since we also need to process ordinary documentation files (say an FAQ) that involve lots of text and occasionally graphics, Tex-info syntax is to be preferred over XML-style markup (well – actually *anything* is preferable over XML-style markup for lots of human digestible text) if at all possible. c) We need to generate documentation from introspection tools (e.g. bseautodoc.c spews out documentation for object properties, signals and channels). d) The source code comments should use the same markup style that is used for ordinary documentation and design documents (e.g. the Beast FAQ or the Beast Quick Start Guide). e) We need to be able to generate decently looking HTML for the Beast website. f) We need to be able to generate manual pages – nothing too sophisticated, covering roughly the feature set of man(7) on Linux is more than sufficient. g) We need a runtime documentation format with support for “live documentation”, this was one of the reasons to go with GtkTextView in the original documentation. Basically “live” here means, runtime documentation displayed by a running Beast instance, e.g. triggered from the “Help” menu, should be able to provide links/buttons/activators that can trigger the running Beast instance to play back documentation example songs. h) Taking (b) to the extreme, writing documentation should be as easy as possible, maybe even as easy as writing documentation with OpenOffice Writer and exporting it as HTML. Of course, this would conflict with (g)… or – would it? i) Automatic external cross-linking should be supported, e.g. to all the documentation sections offered at http://developer.gnome.org/doc/API/. Especially within the reference documentation, automatic link markup would be good to have, without authors having to dig up links or add markup to the relevant keywords. j) The tool-chain required to build documentation should be available from a reasonably standard system, such as the current Debian stable. k) Something similar to the custom macro definition feature of Tex-info as described in The Beast Documentation Quest (1) should be offered by the new documentation system as well if at all possible.

The above requirements at hand, i looked around quite a bit to find a documentation tool that would allow me to fulfil all or most of them. It turned out that (a) and (g) definitely were amongst the harder ones, while (i) was mostly solved already by the old documentation system with regards to link indexing. There, we used a Perl script to extract various kind of (source code identifier, developer.gnome.org/doc/API/-link) tuples from developer.gnome.org. This scrip even covered enum values, something that gtk-doc does not generate own index lists for. One of the next things i was to find out, was that many tools don’t deliver HTML output as nice as that generated by gtk-doc. But then, even the very long-term plans for gtk-doc do not include extension towards C++ parsing… In general, C++ parsing isn’t one of the tasks easily covered. As far as i could figure, there is no decent free C++ parser (including comments) usable for documentation purposes other than Doxygen. Luckily, Doxygen is a reasonably standard tool (j) and also supports ‘@’-style markup that looks Tex-info alike (b) and generates documentation in multiple output formats (amongst them HTML and manual pages). Unfortunately i could not get the Doxygen HTML output to look anywhere close as decent as gtk-doc generated HTML pages. That included attempts at sophisticated CSS modifications and even hacking it’s documentation generation abilities. At closer inspection, Doxygen also did not fully fit what i had in mind for a general purpose documentation tool to cover, i.e. (b), (c) and (k). It should be noted however that Doxygen supports link index imports in a Doxygen specific format that can be used to enable automatic external linking (i) with some effort.

]]>https://testbit.eu/23022006-the-beast-documentation-quest-2/feed/521.02.2006 The Beast Documentation Quest (1)https://testbit.eu/21022006-the-beast-documentation-quest-1/
https://testbit.eu/21022006-the-beast-documentation-quest-1/#respondWed, 22 Feb 2006 08:04:52 +0000http://blogs.gnome.org/timj/2006/02/22/21022006-the-beast-documentation-quest-1/[...]]]>Around 2002, we (the beast project) started to put together the various bits and pieces for a makeinfo based documentation system. We ended up using texinfo markup because it’s not as hard to read and write for humans as XML is, and you can easily define your own macros to aid you in semantic markup. E.g. if you wanted to markup a menu item path as such, you simply define @menu{} to mark things up as desired and then use it like: Use @menu{/File/Save As...} to save a files with arbitrary names. Documentation was needed mainly in three formats: HTML for the website, manual pages for the executables, and in a specific (.markup) format for a GMarkup-based parser that served as a front end to GtkTextBuffer text and tags. This .markup format is actually what Beast used (and currently still uses) as its primary documentation which can be displayed by GtkTextView widgets (.markup files contain tag definitions that define GtkTextTags with specific property settings and tag spans that apply these to text regions). So to generate documentation, we basically did (simplified for brevity): 1) write doc.texi with texinfo markup, 2) generate XML markup from doc.texi with makeinfo --xml, 3) generate the target formats with xsltproc [markup.xsl|html.xsl|man.xsl]. And for display, we did: 4) parse .markup file at runtime with GMarkupParser, 5) create GtkTextTags and adjust their properties from the parsed tag definitions, 6) create a GtktextBuffer to be displayed by a GtktextView, and apply the GtkTextTag spans. On top of all this, we parsed our source files with a perl script to extract source code documentation, and generated .texi files from that. We used a perl script instead of using gtk-doc, because it was hard to use back then and because we wanted to use texinfo markup instead of SGML markup for our documentation (the escaping required by SGML/XML really sucks for source code sequences). So far so good. But then, roughly 2 years ago, we started to get serious problems with our documentation system, basically due to instabilities of the --xml mode of makeinfo. Other annoyances were that we couldn’t document our newly added C++ code, and that GtkTextView/GtkTextBuffer, albeit having a rich set of markup facilities, has no support for <table/> markup in any form (and probably never will). That, and a bit of maintenance lack in our .xsl files led me to look around for suitable alternatives…

]]>https://testbit.eu/21022006-the-beast-documentation-quest-1/feed/025.01.2006 Allocation debugging with GLibhttps://testbit.eu/25012006-allocation-debugging-with-glib/
https://testbit.eu/25012006-allocation-debugging-with-glib/#commentsWed, 25 Jan 2006 12:49:47 +0000http://blogs.gnome.org/timj/2006/01/25/25012006-allocation-debugging-with-glib/[...]]]>I just completed reworking the GSlice and g_malloc() memory debugging hooks. With this, GLib-2.10 (or the next GLib-2.9 development version due this weekend) will (without recompilation) support G_SLICE=always-malloc and G_DEBUG=gc-friendly. These cause all GSlice allocations to go through g_malloc() and g_free() instead and cause all released memory to be memset() to 0. This helps memory checkers to trace pointer lists and can be used to debug g_malloc()/g_free() and g_slice_new()/g_slice_free() allocation code to find and plug memory leaks:

]]>https://testbit.eu/25012006-allocation-debugging-with-glib/feed/120.12.2005 Tuning GSlice memory consumptionhttps://testbit.eu/20122005-tuning-gslice-memory-consumption/
https://testbit.eu/20122005-tuning-gslice-memory-consumption/#commentsTue, 20 Dec 2005 13:22:46 +0000http://blogs.gnome.org/timj/2005/12/20/20122005-tuning-gslice-memory-consumption/[...]]]>The other day, Tommi Komulainen pointed out to me that GSlice is using more memory than memchunks for him after bootup of the N770. Now, GSlice is supposed to be faster than memchunks, yes. And its supposed to scale far better across multiple threads. In the long run it is also meant to be more efficient in terms of memory consumption. However, that is mostly due to the fact that many code portions which use GMemChunk are keeping their own never-freeing trash stacks. Such code should now be migrated to the GSlice API and the trash stacks should be removed. GSlice maintains its own working set memory in per-thread trash stacks. Home grown trash stacks will just waste memory and clutter up the cache lines. Also, using separate GMemChunks prevents chunks of equal sizes from being shared across a program, this again wastes memory and clutters up the cache lines. Because of the long-term wastage that GMemChunk application tends to build up, significant memory savings from GSlice are actually only to be expected for longer running programs which certainly is not a scenario met directly after N770 boot up That being said, the original check-in of the slab allocator in the GSlice code does behave a bit greedy actually. Basically, it allocates a new page (4096 bytes on IA32) per every different size, memory chunks are allocated at (chunk sizes are aligned to 8 bytes on IA32). That means, initially allocating 8 + 16 + 24 + 32 + 40 bytes in separate chunks does require opening up of 5 caches, so it uses 5 * 4096 = 20KB already. I’ve tuned the most recent CVS version now to do more economical caching, so the above scenario now ends up at roughly the power-of-2 sums of 8*8, 16*8, 24*8, 32*8 and 40*8, which is approximately 1.6KB and can all be allocated from a single memory page. While this is a significant memory saver, it also does have some performance impacts. However, in all test scenarios on my machine, the GSlice performance didn’t drop by more than 5%. That’s probably bearable, considering how significant the savings are.

]]>https://testbit.eu/20122005-tuning-gslice-memory-consumption/feed/214.12.2005 Never reject a feature!https://testbit.eu/14122005-never-reject-a-feature/
https://testbit.eu/14122005-never-reject-a-feature/#commentsWed, 14 Dec 2005 04:48:08 +0000http://blogs.gnome.org/timj/2005/12/14/14122005-never-reject-a-feature/[...]]]>I didn’t mean to engage in the flamage about the printer dialog or usability design issues with Gtk+’s new file chooser, but i do feel that a few things should be pointed out in this context nonetheless. In particular, I’d like to respond to Dave Neary (bolsh), quote: “Linus is irrelevant”. There’s an important thing to note here: Linus opinion on GNOME is relevant!
For a single very important reason: Linus is a GNOME user.
Every GNOME user counts. Without exception. Yes, i mean it.
This is the reason we care about accessibility even though it focusses on a user group that can be regarded a minority by sheer usage figures. If you start to make decision only for an assumed “majority of users”, you are going the wrong path with a probability so high you may as well call it a guarantee.

Firstly, programmers are notoriously bad at assuming what the “majority of users” want, and often even what that “majority” is. As Linus put it: “the _vast_ majority of people are part of a specific minority when it comes to something”. As a result, it’s often very hard to tell what your users want or even who they are. The only cure from that is listening to your users, their requests and – if required – poll feedback. If you think these are exclusive programmer problems, think again. Without a specific case study at hand, guesses by usability experts usually aren’t any better. And marketing experts face the same problems, they also need to figure who their (possible) users (customers) are and what they could (possibly) want. The main difference between marketing people, usability experts and programmers is that the former two groups actually make evaluations and studies, they test or ask users and try to find out about their needs and expectations. They do the “feedback polling” just mentioned. (If your project allows for feedback polling to be carried out as a proper usability study, that is always better than just asking the user what he thinks he might need.)

Secondly, disregarding any group of users by marking them as irrelevant is a guarantee for bad perception. There is no way around this, you have – by definition – just made sure that some users are going to be dissatisfied and feel mistreated. Now think about your own reaction when you feel that way. Do you shut up? Do you engage in improving the particular project? Do you try to work out a consensus with someone attempting hard at ignoring you?

Thirdly, disregarding any user needs is not the logical thing to do from an impetus perspective. For that, think of the user request as a force towards a specific direction. Arguing against the request can be viewed as a resistance to the force. As a programmer or project leader, you’re in control of that resistor. If you’re clever, you try to direct that force to work for you, e.g. “We could integrate feature XXX if we had a patch that integrated it according to specification XXX-spec.” The force may be not big enough to be utilized, i.e. the user walks away; fine, without knowing a person really well, you couldn’t have told its strength in advance anyway. Or it might be big enough, you get your patch (maybe from the user directly or maybe by him hiring someone to do the work, etc), you work out the details and you can integrate it into the project. In the end, everyone wins that way (I’ll get to why this doesn’t need to worsen usability).

Alternatively, you can be ***not-so-clever***: You could argue that the feature will never be integrated, for whatever reason, and stand by that point. That is effectively putting the resistor antipodal to the force. Be assured, it’ll hurt both sides no matter how polite you are since this situation strives for conflict maximization.
To be fair, this is a simplification to some extend, sometimes the user simply needs to be told the proper way to achieve something or be kindly reminded of the manual. I have faith in the ambitious reader to tell this difference though

As an interesting side note, let me tell you that the not-so-clever branch of the impetus view can explain forking amongst other things. Consider a force (the user(s)) significantly stronger than the resistor (the blocking developer). There are basically three possible outcomes of this:
1. Argumentation goes on until the developer gives up. Depending on the project structure, various things can happen, the developer might resign, or a patch gets past him by approval of other project developers. The user may (longer term) even become a more active contributor.
2. If the developer is in full control of the project and doesn’t give up, the user can re-aim his energy. He may provide significant contributions to a competing project (maybe a proprietary one), or he may direct his energy against the project by generating bad PR and spreading FUD. Now remember that we assumed above that this force was significant, measured in relation to the developers energy. Significant bashing, hindering the project in various ways, will be the result and can be observed for quite some project/user combination.
3. Let’s say our user who has a significant urge to see his problem XXX solved is technically skilled. He might start his own program/project which solves XXX. However if the project is large enough, a fork will often provide the only viable alternative. So if you ignore enough users especially over topics they deem important enough (they don’t need to appear important to you), you increase likeliness of a project fork, or maintenance of a customization patch set for your project or similar variants thereof.

Now i do realize that not everyone in the GNOME camp wants to ignore and upset the majority of users or user minorities (be warned that “user minorities” may still represent a strong force
So let me plead that we attempt to reduce this where it comes up nevertheless. As things stand, we have too many dissatisfied users currently and can observe the various outcomes described above. To address this, we simply need to strive for fulfilling more user requests, e.g. by accepting more patches that have been rejected as “featurism”, and by re-enabling or exposing existing features (gconf-editor gives you a hint on what could be done already if we allowed more configurability).
And let me side-line with Linus on trading features for usability. Basically, this can’t be done, because usability can only be a second grade concern. If you don’t have features or leave them out, you’ve ruined your usability right away, not improved it. If you have usability problems because of many features, yes that is a serious concern but still second grade. There are various ways at dealing with this, for instance by improving the structure of your preferences or by introducing user levels (e.g. early versions of Nautilus did this) or “Advanced” features (take the mozilla tab extension preferences as an example) or a number of other measurements. The reason there is not “The Definitive Answer” to this currently, is that usability in various aspects is still a very active research topic (let’s talk about this again 100 or 200 years from now). Taking features out is not amongst the set of these usability measurements, because it’s essentially a first grade decision about the program scope and solution space constituted by user desires, so it will definitely get you into the problem chain presented under “Thirdly,” above.

In essence:
In order to avoid various kinds of problems for your users and project, NEVER REJECT A FEATURE but either implement what’s requested or ask for an implementation (nothing simpler than just saying: “We accept patches.”)

Up
date:PBor replied in his blog and suggested a title change for this post to “Never reject a feature request!“, which I much agree with in retrospect.

]]>https://testbit.eu/14122005-never-reject-a-feature/feed/802.12.2005 Landing g_slice_new()https://testbit.eu/02122005-landing-g_slice_new/
https://testbit.eu/02122005-landing-g_slice_new/#commentsFri, 02 Dec 2005 05:56:15 +0000http://blogs.gnome.org/timj/2005/12/02/02122005-landing-g_slice_new/[...]]]>I finally managed to finish up the new g_slice_*() allocator for GLib CVS and commit it. For most platforms, it should be a lot faster than malloc(), and on my machine it saves 20-30% in performance over allocating with mem-chunks. But most importantly, it does share equally sized chunks across a program, which mem-chunks didn’t, so they opened up several independent heaps of equally sized chunks, scattered all over the program and causing large wastage. I would like to thank Matthias Clasen for an initial version of the magazine cache and Stefan Westerfeld for helping me with optimizing the common code paths.

]]>https://testbit.eu/02122005-landing-g_slice_new/feed/208.11.2005 Trust Me! Sorry, No.https://testbit.eu/08112005-trust-me-sorry-no/
https://testbit.eu/08112005-trust-me-sorry-no/#commentsTue, 08 Nov 2005 03:23:59 +0000http://blogs.gnome.org/timj/2005/11/08/08112005-trust-me-sorry-no/[...]]]>Often i have heard “Just trust me on this!” or similar requests during a technical discussion, and even though I would prefer to please the other person, i simply must say “No” to this kind of request, in bugzilla, on mailing lists, on IRC or even face-to-face discussions. “Trust Me” simply is not a valid technical argument. Now don’t get me wrong, i don’t want to attack anyone personally, and on the social level, you probably want to behave the other way around more often than not (most girl friends don’t appreciate “I won’t trust you.” as far as my experience goes at least), but not when you’re trying to maintain reproducible results, often according to scientific method. Many programming errors are based on false untested assumptions and assumptions can be false for an unbelievable variety of reasons, basing on “Trust Me” is just one of them Conversely, I certainly would love everyone in the world to trust my person on every (technical) fact i present, but over time i had to figure that i can’t even trust myself on every measure, so it would be ridiculous to ask others to do so. Completely valid, long standing assumptions can be broken by such simple things as a package upgrade, a software fix, processor variant switch, hard disk fill level, a timer wrap around or specification changes that i haven’t heard of ;-[] And that was only warming up the list. So over the years i have tried to make a habit out of basing every programming or design decision i make on a solid basis of facts, each of which i have checked at least once, heard a sound argument for or seen a reasonable proof for (and usually should be able to dig up the check for it, or relevant information resource). This does render “Trust Me!” a completely irrelevant statement in technical arguments though, and with this blog entry i want to apologize for every time i had to respond “No” to such a request and plead for an attempt at understanding why i do this. I don’t mean to offend anyone with it, but to improve the software I’m producing for other people; so to a certain extent, i feel i have the obligation to question the reasonings and facts I’m supposed to operate with. Thank you for reading this.

]]>https://testbit.eu/08112005-trust-me-sorry-no/feed/301.11.2005 My brand new toyhttps://testbit.eu/01112005-my-brand-new-toy/
https://testbit.eu/01112005-my-brand-new-toy/#commentsTue, 01 Nov 2005 08:18:25 +0000http://blogs.gnome.org/timj/2005/11/01/01112005-my-brand-new-toy/[...]]]>A few days ago, stefan demoed x2x to me, and i’ve fallen in love with it since. If you’re not regularly working with multiple machines (like a desktop and a laptop next to it) it probably won’t be of much use, but if you do, this little program allowes you to forward mouse and keyboard events from one X server to another simply by leaving the screen with the mouse (it also allows relaying of the PRIMARY selection between displays). So i’ve put my old laptop next to the desktop screen now and use it as some kind of screen extension, mostly for IM/IRC. Here’s the ssh command i’m using from the desktop machine to spawn the forwarding without doing insecure xhost hackery:

Note that the $DISPLAY is evaluated by the ssh shell and will expand to the display of the forwarded X connection, instead of the display value x2x is started with.

]]>https://testbit.eu/01112005-my-brand-new-toy/feed/423.09.2005 GtkRC Matching During Gtk+ Startuphttps://testbit.eu/23092005/
https://testbit.eu/23092005/#respondFri, 23 Sep 2005 17:38:30 +0000http://blogs.gnome.org/timj/2007/06/01/23/[...]]]>The other day we had been looking some into Gtk+ startup time and found that adding lots of widgets to a window hierarchy in combination with extensive gtkrc file usage, can result in RC style matching showing up with significant figures in the profiles. The basic RC style match logic in Gtk+ already had been optimized a great extend in 1998, and with recent patches from Matthias Clasen, the complexity of wildcard string matches in GPatternSpec was furtherly reduced.
That’s as far as optimizing the code goes, however significant optimizations can also be gained from modifications to the set of RC style match rules in the gtkrc files. So in order to enable everyone to performance optimize their gtkrc file rules, we have compiled a list of optimization directives to apply to existing gtkrc files:
[Optimizing RC style matches in gtkrc files]

It’s “Bye, bye hard disk, thanks for doing a great job!”-time again. The main disk of my development machine (IBM Ultrastar, Nov 2000) died somewhere between 05:00 and 08:00 last night. I hadn’t been keeping too much stuff besides the main system on it recently, simply because 34G are filled up pretty quick these days. Nevertheless, there’s still the home dir of my local development account (a scratch account) on it. And allthough that has mostly temporary rubbish in it, besides my current development files, it does keep the settings of a pretty fine tuned desktop configuration…
So i went out earlier today to get a new hard disk (a fast, quiet and cheap P-ATA from Samsung – i’m slowly replacing all SCSI devices here) and put it into my development machine, taking proper care of grounding and unplugging everything appropriately. Believe it or not, once i turned everything back on, the BNC input of my 21″ monitor died. Double checked graphics card, connections etc, tried the monitor on the laptop… nothing. That input stage simply burned out. Bummer. Ok, so i’m using the D-SUB input now (luckily that monitor has two inputs) to copy over the disk contents to the new drive. While i’m writing this, that hot ol’ SCSI lady (you can’t touch it while it’s running, it’s way too heated) is making an awfull lot of noise on some sectors, kinda scary since i’m still copying some relevant development files.
Always remember, skip you not the daily backup!

]]>https://testbit.eu/08082005/feed/006.08.2005 Rapicorn PNG Loaderhttps://testbit.eu/06082005/
https://testbit.eu/06082005/#commentsSat, 06 Aug 2005 18:00:37 +0000http://blogs.gnome.org/timj/2005/08/06/06082005/I just implemented the PNG image loader for rapicorn, albeit without animations so far. Regarding the Animated PNG specification, i actually got word from Stuart Parmenter in response to my recent blog entry, that the spec should be fixed up and finalized in the near future. So there’s still hope for a low-complexity animation format.

]]>https://testbit.eu/06082005/feed/101.08.2005 Working At Imendiohttps://testbit.eu/01082005/
https://testbit.eu/01082005/#respondMon, 01 Aug 2005 19:10:06 +0000http://blogs.gnome.org/timj/2005/08/01/01082005/[...]]]>Today, i started working at imendio, we’re doing some pretty interesting hacking here, so i hope the job is going to be fun.

]]>https://testbit.eu/01082005/feed/031.07.2005 – Why Rapicornhttps://testbit.eu/31072005-rapicorn/
https://testbit.eu/31072005-rapicorn/#commentsSun, 31 Jul 2005 10:08:51 +0000http://blogs.gnome.org/timj/2005/07/31/31072005-rapicorn/[...]]]>I have been asked some about rapicorn recently, so…
In spring 2005, i started working on a gnome-canvas replacement for beast. A bunch of GUI/toolkit design ideas that accumulated over the last few years have influenced the design of this. For a change, i coded this up in C++ so i am not forced to produce the large amounts of boilerplate code that GObject requires. It is only with C++, where typesafe interface lookups for composite widgets can be implemented. That is, a container holding a child of a specific interface (say a box containing a text entry which implements the Text interface) integrates that interface into its public API via:

Text &text = container.interface<Text>();

This can be extended to allow passing in widget ids, for instance to tell apart the buttons of a dialog:

Button &button = file_selector.interface<Button> ("cancel-button");

And to access the text of the cancel button:

Label &label = file_selector.interface<Label> ("cancel-button");

The core idea is to build larger widgets out of micro widgets (very simple things like frame, label, table, image, rectangle shader, etc.) by writing concise xml statements. This may look like so:

This would be a button, packed into a table at 50-51 horizontally and 59-60 vertically. Also, the button is expanded horizontally within the table container and contains a label with translated text. These xml style widget definitions were originally developed for Gtk+ code and can be found as “radgets” in recent beast releases. The notation is powerful and conscise enough that nearly all beast dialogs could be defined in a single xml file.
Another major design aspect is strict seperation of model, view and controller logic, down to such simple things like toggle buttons. Most popular toolkits out there already allow seperation between model and view, but fail short on seperating the controller logic and making it pluggable (customizable). Most often this causes code more complicated than neccessary, where the view and controller logics are closely tied up, and result in hacks or code duplication in third party products to change controller behaviour as desired. Turning controller functionality into its own object instance which may be customized or replaced by the toolkit user, will help to reduce third party hacks and increase structure of the toolkit code.
Last but not least, an often heared request for beast was changing its outlook, so it looks more like “cool” music applications found under windows: (AU), (RE), (CU). This will involve working more with images, font rendering, alpha blending and dithering for standard GUI components than is intended with stock Gtk+, so this kind of featurism is also supposed to end up in rapicorn instead.
That being said, rapicorn hasn’t left initial design phase yet and much of what is there or will be there soon may not survive future releases because we’re still testing out new ideas and concepts. Semi frequent releases however will be made available at the rapicorn ftp directory, mostly in response to someone asking me to put up a new tarball.

]]>https://testbit.eu/31072005-rapicorn/feed/226.07.2005 Usability Of Dnyamic Menushttps://testbit.eu/26072005/
https://testbit.eu/26072005/#respondTue, 26 Jul 2005 20:13:10 +0000http://blogs.gnome.org/timj/2005/07/26/26072005/[...]]]>The user interface design newsletter from last years july has an interesting article on Adaptive Menu Design. The bottom line is:
Do not use dynamic menus (i.e. hiding or moving menu items), since people learn by spatial location.
For quick reference, and for everyone too lazy to read up on menu design, Beasts usability page menu section has a concise list of design directives to comply with for proper menu construction.

]]>https://testbit.eu/26072005/feed/023.07.2005 Images In Rapicornhttps://testbit.eu/23072005/
https://testbit.eu/23072005/#commentsSat, 23 Jul 2005 11:52:57 +0000http://blogs.gnome.org/timj/2005/07/23/23072005/Implemented images in rapicorn. So far, the various pixstream formats produced by gdk-pixbuf-csource can be deserialized and displayed, here’s a screenshot. The next item on the list is supporting libpng natively, ideally i’d add support for simple animations along the way, but the Animated PNG specification doesn’t seem to be moving forward anymore…

]]>https://testbit.eu/23072005/feed/317.07.2005 Der Hummelflughttps://testbit.eu/17072005/
https://testbit.eu/17072005/#respondSun, 17 Jul 2005 10:22:21 +0000http://blogs.gnome.org/timj/2005/07/17/17072005/[...]]]>Been busy the last few days with creating a povray-rendered movie as part of a university project. Rendering can be very time consuming depending on the scene, so the resulting movie really is short, close to 2 minutes 30. Allthough we employed various measures to reduce rendering time, like creating a camera effect that only requires a portion of the resulting frame to be rendered (compound eye), or by creating a large image with gimp for the credits section and turn it into movie frames by moving a viewport along its height.
Anyways, here you go, The Bumblebee Flight:Der Hummelflug (39M)
And a German summary about the creation process:Der Hummelflug (Documentation)

]]>https://testbit.eu/17072005/feed/006.07.2005 Rambogotchihttps://testbit.eu/06072005/
https://testbit.eu/06072005/#respondWed, 06 Jul 2005 11:49:06 +0000http://blogs.gnome.org/timj/2005/07/06/06072005/Thanks to everyone who submitted hackergotchi mockups of my original image!
Here is the resulting rambogotchi, you can stop contributing now

]]>https://testbit.eu/06072005/feed/005.07.2005 Hackergotchihttps://testbit.eu/05072005/
https://testbit.eu/05072005/#commentsWed, 06 Jul 2005 07:25:31 +0000http://blogs.gnome.org/timj/2005/07/06/05072005/[...]]]>Help! I need artistic help, my problem is: i’m faceless! At least as far as the planet is concerned…Here is head shot of me which needs some editing to remove the background and add a drop shadow effect, in order to be accepted as a proper hackergotchi by jdub. If anyone is capable of doing these edits, please help out with some image surgery, so i get a hackergotchi identity

]]>https://testbit.eu/05072005/feed/330.06.2005 More Command Line Web Blogger Hackinghttps://testbit.eu/30062005/
https://testbit.eu/30062005/#respondThu, 30 Jun 2005 17:42:17 +0000http://blogs.gnome.org/timj/2005/06/30/30062005/More hacking on diary-publish.py, the command line web blogger script. Since http://blogs.gnome.org/timj is currently down due to the gnome server move, i added a simple html writer to it that allowes previewing of the marked up diary with an html browser before posting them.

]]>https://testbit.eu/30062005/feed/029.06.2005 Quick Random Number Generatorhttps://testbit.eu/29062005/
https://testbit.eu/29062005/#respondThu, 30 Jun 2005 17:02:20 +0000http://blogs.gnome.org/timj/2005/06/30/29062005/[...]]]>Majorly improved the speed of rectangle shading in Rapicorn, here’s a quick-and-dirty random number generator which is still good enough for dithering:

This has been the major speed-up-knob, other aspects were as confusing as profiling often gets, e.g. -ffast-math actually slowed down the rectangle shader by 13%.

]]>https://testbit.eu/29062005/feed/028.06.2005 Open Wikishttps://testbit.eu/28062005/
https://testbit.eu/28062005/#commentsThu, 30 Jun 2005 13:34:20 +0000http://blogs.gnome.org/timj/2005/06/30/28062005/[...]]]>Wiki access. When i come across a minor error or easy-to-fill spot on wiki sites like Wikipedia, i often take the time to edit the content appropriately. I can’t be bothered to create an account for such minor edits though. Especially not if i’m unlikely to ever return to the webpage since creating an account means generating a new password (sometimes even an untaken username), and taking notes about the site and the newly generated account data.
So wikis meant to absorb contributions from occasional visitors need to accept anonymous edits, even if that means regular anti-spam edits have to be carried out.
Consequently, beast has such an open wiki. And yes, i’m deleting spam including history from it and keep a spam word filter updated on a daily basis, because i do think open access is worth it.

]]>https://testbit.eu/28062005/feed/123.06.2005 How Nature Forms Structurehttps://testbit.eu/23062005/
https://testbit.eu/23062005/#respondThu, 23 Jun 2005 20:06:20 +0000http://blogs.gnome.org/timj/2005/06/23/23062005/[...]]]>How’s nature forming interesting patterns such as the surface of sea shells, or simply telling apart the places to grow a head and a leg?
The mathematical explanation describes this by the interaction of an activator and an inhibitor which are produced in a concurrent, mutually influenced reaction with varying diffusion properties. Basically, this allowes local activation (activator) with lateral inhibition (inhibitor), i.e. an attribute emerges at a certain location and is suppressed in the near surroundings, but may be repeated in surrounding regions farther away.
The result is: structure! Structure, which emerges out of a homogeneous cell mass where all cells contain the same operational program (gene code). This page has a good explanation on the reaction details. By tweaking various properties of the fundamental two differential equations that describe this process, many aspects of pattern formation can be explained. Large gradients (inhibitor concentrations) can be used to form global coordinate systems which are suitable to control whole body formation, repeated small gradients (activator concentrations) allow for oscillation (pigment patterns of sea shells) or near optimal attribute coverage on surfaces (hairs).

]]>https://testbit.eu/23062005/feed/020.06.2005 Command Line Web Bloggerhttps://testbit.eu/20062005/
https://testbit.eu/20062005/#commentsMon, 20 Jun 2005 21:32:58 +0000http://blogs.gnome.org/timj/2005/06/20/20062005/Have been hacking on the command line web blogger today and learned python on the way, so can you read this? Posting works already, q.e.d. However, NewsBruiser is giving me a hard time with not accepting editPost() requests. So in effect, i’ll not be able to update this entry later on which sucks.

]]>https://testbit.eu/20062005/feed/115.06.2005 Server Died And Lots Of Bugshttps://testbit.eu/15062005/
https://testbit.eu/15062005/#respondMon, 20 Jun 2005 21:14:20 +0000http://blogs.gnome.org/timj/2005/06/20/15062005/[...]]]>Finally reconnecting… since my server died earlier with a reproducible kernel BUG in the resierfs journaling code of a stock 2.6.10 (journal.c:2825). After poking around a bit, i decided to copy everything over (more than 100 Gigs) and move to ext3. Especially since i recently downgraded from 2.6.11.* because i ran into two other scenarios of filesystem corruptions under reiserfs (on different disks btw). So i simply don’t trust the reiserfs code anymore and am migrating all my machines to other file systems bit by bit. Sent off a bug report to the reiserfs people and ran into another bug in mc while backing up my files.

]]>https://testbit.eu/15062005/feed/014.06.2005 Ideas For Beasthttps://testbit.eu/14062005/
https://testbit.eu/14062005/#respondMon, 20 Jun 2005 21:14:06 +0000http://blogs.gnome.org/timj/2005/06/20/14062005/[...]]]>Sat down with stefan today to evaluate GarageBand to get some ideas for improvements and neccessary additions in beast. The number and types of loops it offers out of the box (actually we also had Jam Pack 1) is simply thrilling. Cooking up a rich dance music background is a matter of minutes. We got lots of new GUI ideas out of it, most of which ended up in the beast TODO, and we got more dedication for extending the stock instrument set shipped with beast.

]]>https://testbit.eu/14062005/feed/013.06.2005 Intersection-Free Polygon Unfoldinghttps://testbit.eu/13062005/
https://testbit.eu/13062005/#respondMon, 20 Jun 2005 21:12:59 +0000http://blogs.gnome.org/timj/2005/06/20/13062005/[...]]]>Today we’ve been having a visit of Ileana Streinu at our university, where she gave a very interesting talk about intersection-free polygon unfolding in 2D space and related topics. One idea struck me as particularly interesting. A bit oversimplified, it is the reduction of available degrees of freedom in polygon expansion until only one degree of freedom is left for the expansion algorithm to operate on. An older but somewhat shorter explanation can be found in this paper.

]]>https://testbit.eu/02062005/feed/001.06.2005 GUADEC: On Atomic Reference Countinghttps://testbit.eu/01062005/
https://testbit.eu/01062005/#respondMon, 20 Jun 2005 20:27:32 +0000http://blogs.gnome.org/timj/2005/06/20/01062005/[...]]]>Back from GUADEC, still a bit tired though. Thanks to all the helpers who made this happen! It was nice to see all those people again and talk to them. Sat down with wim to discuss atomic reference counting issues for GObject. We’ve figured most of the issues, but GClosure is still giving a bit of a headache, because its reference count is part of a bitfield.

]]>https://testbit.eu/01062005/feed/026.05.2005 BEAST/BSE Version 0.6.6https://testbit.eu/26052005/
https://testbit.eu/26052005/#respondMon, 20 Jun 2005 20:25:48 +0000http://blogs.gnome.org/timj/2005/06/20/26052005/BEAST/BSE version 0.6.6 is out, managed to throw it out before GUADEC despite the long list of must-fix items i had for this release. Will be heading to stuttgart for GUADEC6 tomorrow.

]]>https://testbit.eu/26052005/feed/023.05.2005 Adjusting The Beast Dialogshttps://testbit.eu/23052005/
https://testbit.eu/23052005/#respondMon, 20 Jun 2005 20:25:40 +0000http://blogs.gnome.org/timj/2005/06/20/23052005/[...]]]>Ok, no new entry in the last few days (3.5 years). Have been adjusting the beast dialogs the last few days as well as patching intltool to support full translation markup of strings. That is, allow multi-line markup of strings with functions, e.g. (_ “A ” “string”), same goes for (Q_) and (N_). The patch now needs to replace the scheme string hackery currently implemented in intltool which is used for script-fu and for script-fu only (because it uses _”A string” as markup, which isn’t even valid scheme syntax according to R5RS). In other news, it was Stefan’s birthday today, so we all (his mom, Dirk) went out for dinner and had some nice chitchat.