Yesterday I spent all day working on the reincarnation of the status monitor. Basically what this means is that I started from scratch, worked out the ideal architecture based on past experience and starting reassembling the extension by copying code from trunk and the old status monitor back in (reviewing it and improving it wherever needed).

I’m far from done, for example I haven’t yet started on implementing the ideas for the auto-updating status tree, but it’s starting to look pretty descent. Besides obviously removing a lot of crud by refactoring, here’s what major things have been done so far:

Reimplemented the status monitor, but this time using gio.GFileMonitor. Not only does this mean that NautilusSvn will run on all systems that support GFileMonitor (e.g. *BSD) but the resulting code is even prettier!

Moved to using anyvc for the VCS Abstraction Layer. This library was originally developed for usage with PIDA, a Python IDE. This means that there’s now free emblem handling for at least Subversion, Bazaar, Git, Monotone, Mercurial, Darcs. So no matter what VCS you use you will always see pretty emblems!

If you’re interested you can take a look at the resulting code. Once I get this branch into shape I’ll start making development snapshots so everybody can play around with it.

In other news thanks to Vadim Peretokin I figured out what was causing the Launchpad buildbot to fail on building NautilusSvn, turns out I was missing a few build-dependencies that were required because of commands called from the Python distutils script (pkg-config and gtk-update-icon-cache). Thanks Vadim!

I just created this diagram of the envisioned overall architecture of NautilusSvn and thought it would be useful to share. It doesn’t completely represent the currently implemented architecture though it is quite close.

The D-Bus service consists out of a status monitor which uses inotify to monitor the filesystem and a status cache to speed up status requests etc. The status monitor will notify all registered clients of any interesting actions, so information is pushed to clients. Note that it may be possible for clients to use the VCS abstraction layer directly and not have to pass through the D-Bus service if they so prefer.

I’ve been hanging around the Nautilus IRC channel for quite a few years already and have gotten quite familiar with the devs. Every single one of them is a great individual and always willing to lend a helping hand.

I thought I’d ask Alexander Larsson, a Red Hat employee and the lead developer for Nautilus and other projects including GVFS, for his insights as to why initial status checks take so much longer than consecutive ones. I already had a general idea as to why (mostly the idea of caching) but was not familiar with the details and quite interested in the opinion of somebody I consider a true expert.

The resulting answer was so informative I just had to take the time to share it with all of you. Here’s how Alex explained it:

Well, you know about caching I suppose? At this point we’re talking about the kernel using spare RAM to keep information about whats on the disk.

Say you start with a blank slate, i.e. you have not accessed the filesystem at all. Now say you run stat(“/some/dir/file”). First the kernel has to find the file, which in technical terms is called the inode. It starts by looking in the filesystem superblock, which stores the inode of the root directory. Then it opens the root directory, finds “some”, opens that, finds “dir”, etc. eventually finding the inode for file.

However, on a second access to /some/dir/file it uses the “dcache” (directory cache) which keeps around a set of recently accessed paths like /some, /some/dir, and /some/dir/file. So, it can now find the inode without any disk I/O.

Then you have to actually read the inode data. After first read this is also cached in RAM. So, a read only has to happen once.

I interupted Alex for a moment to ask him how long entries are maintained in the cache.

It depends on what else the system is doing. If you actually start reading the file data, that is also cached (unless you hint the kernel not to do it). In general Linux tries to use all memory that is not otherwise allocated for cache and has a form of least recently used policy for what to throw out. So, if there is any form of memory pressure, the oldest cache info is thrown out. So, reading lots of data is a good way to invalidate caches. Which is why there are ways to hint the kernel that the data read will not be reused.

He continued:

Now, if you look at a HD performance sheet you see pretty impressive performance figures, maybe a disk can read 10MB/s, which surely sounds a lot more than what some itty bitty svn info is. I mean, if your svn status took 1s does that mean it had to read 10 meg of data? The problem is that the read rates is when you read consecutive data from the disk.

Think of the HD like an old record player, once you’re in the right place with the needle you can keep reading stuff fast as it rotates. However, once you need to move to a different place, called “seeking” you’re doing something very different. You need to physically move the arm, then wait for the platter to spin until the right place is under the needle. This kind of physical motion is inherently slow so seek times for disks are pretty long.

So, when do we seek? It depends on the filesystem layout of course. Filesystems try to store files consecutively as to increase read performance, and they generally also try to store inodes for a single directory near each other but it all depends on things like when the files are written, filesystem fragmentation, etc. So, in the worst case, each stat of a file will cause a seek and then each open of the file will cause a second seek. So, thats why things take such a long time when nothing is cached.

Some filesystems are better than others, defragmentation might help. You can do some things in apps. For instance, GIO sorts the received inodes from readdir() before stating them hoping that the inode number has some sort of relation to disk order (it generally has) thus minimizing random seeks back and forth.

One important thing is to design your data storage and apps to minimize seeking. For instance, this is why Nautilus reading /usr/bin is slow, because the files in there generally have no extension we need to do magic sniffing for each. So, we need to open each file => one seek per file => slooooow. Another example is apps that store information in lots of small files, like gconf used to do, also a bad idea. Anyway, in practice I don’t think there is much you can do except try to hide the latencies.

I also asked Alex why Thunar was so fast at loading the same directories, however a few moments later it occurred to me that the reason I thought Thunar was fast might have been because I was opening /usr/bin in it after I had already previously opened it in Nautilus, therefor the seeking and caching Alex talked about had already occurred. Alex responded with a resonant “aha!”. This may well be the reason why so many people say Nautilus is slow compared to X, they’re probably doing the same thing.

He then explained to me how to clear the cache, it turns out all you have to do is execute the following command as root:

sync; echo 3 > /proc/sys/vm/drop_caches

After doing so you’ll see that Thunar will also take quite a while to load up a directory such as /usr/bin, because of the same reason Nautilus does. Another interesting tidbit Alex pointed out was timing gvfs-ls directly. He told me to try out the following commands and stated “you’ll be surprised”.

Note that the first gvfs-ls command took 16 seconds and the latter took 1.5 seconds. He explained that the only difference is that the first one reads the first 2k of each file. With the answer he provided earlier it should be quite obvious why there’s such a difference between the two commands.

Alex ended with the following note:

The real fix for this whole dilemma is to move away from rotating media. I hear the intel SSDs are teh shit. Linus swears by them.

Based on the results so far from the performance poll with a sample of 130 users it seems 38% of NautilusSvn users are not satisfied with the current performance of the extension, while about 43% of the users could be considered satisfied. 19% rated performance as acceptable but it’s a thin line both ways so I won’t count them in either camp. There are very few people that flat out refuse to consider NautilusSvn because of this issue, if I had to take a guess I’d say some 11% ;-)

If I had to place a wager I would bet that most people don’t mind if NautilusSVN takes a while to determine the status for entire working copies (to a certain extent), as long as it doesn’t overheat their CPU, doesn’t consume too much memory and most importantly doesn’t hang Nautilus. If anybody disagrees with this please leave a comment.

However, judging by some of the comments it seems that some people just have extremely large working copies or tend to collect a lot of extremely large working copies in a single project directory. Based on some tests with timing the PySVN status method and the command-line Subversion client I would have to say that there’s really not a lot that can be improved with regards to actual performance. I’m sorry to have to say it, but that’s just the way it is.

Allow me to elaborate.

First, note that on average initial status checks take about 10x longer than consecutive ones. For example, the initial check for the entire TortoiseSVN working copy takes 8309.0079 milliseconds, consecutive ones take 865.9279 milliseconds. That’s 8 seconds compared to 0.8 seconds, I think everybody agrees with me that that’s quite a difference. However, as I see it there’s simply no way to speed up that initial status check. So, say you have 15 working copies the size of TortoiseSVN organized in a single directory, upon entering that directory it would take NautilusSvn some 2 minutes to just figure out the statuses.

Also note that if we want to properly keep working copy and directory emblems up-to-date we’ll also have to recursively register watches on each working copy, initially registering watches using inotify in the case of the TortoiseSVN working copy takes quite a few seconds (I didn’t time it).

There’s still room left to make NautilusSvn more efficient and perhaps a bit more snappy here and there. Especially with regards to the status logic there are still improvements that can be made. Also the implementation of a proper cache will certainly help in making consecutive status checks even faster. But none of this will result in substantial improvements in the area of initial status checks.

So if there isn’t a way to substantially improve performance with regards to initial status checks what can we do? What I know we can do is create the illusion of performance or possibly degrade some functionality. Here’s a few things that come to mind:

Pre-loading working copies. Do the initial status checks when the user isn’t looking. It’s probably a good idea to not do this immediately after booting. We would also have to make sure the computer is not on battery power.

Allow the user to configure to disable NautilusSvn for certain directories.

Do some scheduling tricks and progressively check parts of a working copy. This will also help prevent 100% CPU usage issues for a considerable duration of time (leading to overheating).

Executing the status checks asynchronously

However, let me point out that doing everything asynchronously (i.e. in the background) will only obscure performance issues. Sure, Nautilus wouldn’t hang anymore but NautilusSvn will still be hacking away in the background (possibly causing your CPU temperature to rise to unacceptable levels). Especially when developing I find Nautilus hanging a very useful indicator on whether or not progress is being made.

In the end, irrelevant of the performance issues, what I’m most interested in is having an elegant, flexible, maintainable and robust codebase.

I know that compared to the early months of 2009 it has been a little quiet around the project lately. But don’t worry, we’re not dead! It’s just that Adam is getting married this summer and quite understandably pre-occupied with planning the wedding. I myself simply haven’t had the energy lately to work on the project, there’s some tough issues to solve and I haven’t had any brilliant ideas.

I’d like to take a moment to thank everybody that has contributed to the project so far, from the people providing patches and translations to the people participating in the discussions on the mailing list and on this blog. Note that even though I haven’t yet participated in the discussions on this blog I always read the comments that everybody posts.

For me personally my involvement in this project has been an amazing experience, one that I would like to continue. I consider my responsibility as a project manager to make the barrier towards contribution as low as possible and motivate the community to participate. There’s still a lot that needs to be done to make NautilusSvn more easy for us and other people to develop on and that’s going to be one of my major focus points

The coming months I will be focusing on refactoring NautilusSvn to make several improvements to the overall design which will lead to an overall better application but also improvements in performance and hopefully memory consumption. I’ll be documenting my progress on the Architecture and Code Sprint wiki pages.

Once actual work gets underway snapshots will be regurarly released. In the meantime people will have to make do with the current beta release.

Note that I’m still looking for help in getting the PPA set-up and ready to go for distributing NautilusSvn. The Launchpad buildbot is currently unable to build any packages for the sources I upload and I have not been able to understand from the logs yet precisely what the issue is. Contact me over the mailing list if you think you can help.

UPDATE: Thanks to Vadim Peretokin I figured out what was causing the Launchpad buildbot to fail on building NautilusSvn, turns out I was missing a few build-dependencies that were required because of commands called from the Python distutils script (pkg-config and gtk-update-icon-cache). Thanks Vadim!