Resources in glib

Last week I landed a new feature in Glib that I’ve been wanting to do for a long time: A resource framework. Resources are things that are naturally part of an application or library, but not really normal code. For instance, our code increasingly uses xml to describe user interfaces and menus.

Traditionally these either had to be manually inserted into the code, like so:

Or a file was stored in /usr/share/$application/ and you had to write code to manually find and load the file, and cache it if used often. This is not a lot of code, but it can be tricky as all I/O code needs to handle errors and the external file makes it harder to make the library/app relocatable.

Instead, with resources you store your data file as plain files in your source tree, edit them with your favourite editor, with full syntax highlighting, automatic indentation, etc. Then you reference these files from a resource description file (another xml file) and use glib-compile-resources to compile these into a binary chunk which is linked into the application.

The resource framework then automatically registers all such resource bundles in a global namespace, where you can quickly look up resource data by a resource path. There are API calls to get the data as a direct pointer, as well as streaming data, but in most cases there are helper functions that let you just specify the pathname. So, for instance, you can just do:

Which would handle all the work for you. And while this looks like an I/O operation its really just a hashtable lookup on linked-in data, reading from the (shared, readonly) data section in your executable, so its very fast and safe.

Additionally there are some tricks the resource compiler can do for you. For instance, you can specify that a resource should be compressed, which means that the data is stored compressed, and the APIs uncompress for you automatically. You can also specify that xml files should be pre-processed to strip away whitespace, which avoids wasting time and memory on something that is not useful at runtime.

There is also support for resource:// URIs, which means you can easily reference resource data like icons from e.g. CSS and UI files.

On Linux we use some gcc specific extensions to store the resources in separate ELF sections, which means its very easy to extract resource data from the binaries. Glib even ships with a tool that lets you do this:

24 thoughts on “Resources in glib”

Alex, using gcc constructors for this is a really bad idea, are you sure you got everything right regarding the order in which these constructors are run, and in regards to this constructors when they are used in shared libraries or loadable modules, or even shared libraries which are pulled in by loadable modules?

Using gcc constructors is usually not a solution but a problem in itself.

Lennart: The use of gcc constructors is optional, we also support a manual initialization mode, although that requires your code to have some initialization point, which is not always true for libraries.

As for the ordering issues, I don’t think its really a problem, as the constructors themselves are so trivial. All they do is allocate a chunk of memory and put it on a list. So, there are no actual ordering requirements.

Also, having active code for the registration of the resources might result in quite a hit on performance at startup if a gazillion of small objects are registered by your app and all its shared object.

I think it’s really important to turn this “active” registration into something passive. For example, glibc provides dl_iterate_phdr() which should be enough to implement this entirely passively (with a tiny bit of ELF parsing).

The thing about constructors/destructors is that they appear trivial in their code but seldom are in their effect. And if I hear that you allocate memory in them this causes all my red lights to flash, because this either causes memory leaks if your .so is pulled in via dlopen() (or indirectly via dlopen()), or might result in code not expecting it to access invalid memory. The least you need to do is ensure that shared objects which do this become unloadable…

Lennart: Not only do we have destructors to go with the constructors, we also have test cases for runtime module loading and unloading.

I don’t think using dl_iterate_phdr is a solution, as it only works on glibc, meaning that instead of actually solving any problems we have we just punt them to other platforms.

Also, I don’t think a gazillion objecs are registered. We just register a single pointer to a binary blob. I.E it doesn’t actually try to look into the resource blob at registration time. And this only happens once for each shared library that uses resources (which is not gonna be all of them).

Marc-André:
Self contained applications is a different matter. Resources is not for that. There is no way you can e.g. include another library as a resource, nor is it a good idea to include things other processes needs to read, like docs or desktop files.

If you’re interested in application bundles, see my other work on glick2 for that kind of thing.

There is nothing linux specific about resources, they work fine on Windows, although the gresource tool will not be able to extract resources from win32 dlls.

do you include the functionality to convert png files to in-line pixbufs in the same code so we can handle that the same way as other resources? Right now in Bluefish we convert all icons to C code that is compiled into the binary, so it would be great if there would be a single solution for all these issues.

b.t.w. is there a nice way for backwards compatibility? I would like to keep Bluefish compatible with gtk-2.24 for a while, but at the same time use the new resource system if available.

Oliver:
There is no support for inline pixbufs atm. Its slighly problematic to do in the resource compiler, as it is in glib and doesn’t depend on gdk-pixbuf. It should be possible to do using an external tool though.

GResource is a glib api, and as such works fine in Gtk2. However, the gtk+ helper functions were added in gtk3 only, so you’d have to use a bit more code to actually use the resources.