Birth of a Linux-Distribution

Menu

Monthly Archives: January 2012

We had some time after the initial release to think about where we want to go from here. This is whats on the agenda so far:

KDE SC will for sure see 4.8.x in the next release, that much is clear. Unclear is still, if the kdepim-suite will be updated as well in Debian, so we could see Kmail2 or not. That needs to be taken with a grain of salt. The new kdepim-suite integrates Kmail2 fully in akonadi. The migration to Kmail2 still creates problems with existing accounts, newly created accounts reportedly work fine. But as most everybody has large amounts of mail in numerous accounts, be it pop or imap, that need to be migrated in a clean way. Until that works reliably, the qt-kde-team in Debian thankfully has decided to hold back the new kdepim-suite. That might as well be the case for KDE SC 4.8. I myself would appreciate that.

For the other DEs we ship, I am sure, towo and coruja will work on improving their respective babies XFCE and LXDE, the latter, LXDE, being the newcomer in the team. But so far nothing concrete has been decided there. Users with a wishlist in that respect should post these issues now on our forum, so that there is enough time to work on these.

I myself am looking closely into Razor-qt and a lot of qt-apps to go with it. Razor as a lightweight qt-pendant to LXDE is still in a very early state, but to push it to more users, i might want to ship it at least as a development-release with 2012.1. That depends on how fast it evolves and its too early to make promises about that now. At least, developement happens on a day-to-day base, which is good. Lets see how much improvement can be done in 2 months.

One often asked for flavour over the years is a minimal iso with no X-Server. We are thinking of shipping that maybe with 2012.1. So hold your breath for that.

Besides that, we will ask our users soon to come up with a codename for 2012.1. As you might know, we name our releases after famous rocksongs. (of course pop, punk and all the other flavours of modern day music are included) A famous song is one thing, the other is: can it be visualized and carry a message that relates to us in any way? The 2nd part is up to our art-team. So await our call for action on the forum soonish 🙂

A little over 2 weeks have passed since our first final release, and things are moving along rather smoothly. We shipped no grave bugs, we did forget a few packages though. A lot of work went into the manual in the past weeks and i am proud to say, the english and german files are uptodate again. The manual has also been pimped visualy to a clearer look, a better readable font, and our Corporate Design has been added to it. Also 3 translators / maintainers volunteered to review and update the manual files for their native language. These are italian, greek and polish. Thanks a lot for that! These languages will go online again within a week. If you can help with your native language, please do so!

We had a few reviews, none of them had much negative things to point out. I will write 2 articles for the german print-magazine LinuxUser. One will present the project with its ressources and goals, the other will be based on this blog and will feature the birth of a distribution along the lines of my posts here.

We will present the project with a booth in March at Chemnitzer Linuxtage and in May at LinuxTag Berlin. We will share a booth with Debian at both events, maybe Kanotix will join in for LT Berlin. As a special for LT i had the idea, that we could do a special LT-release, built and released at the booth. Visitors could have it copied to a siduction-branded USB-Stick and carry it home. That should attract quite a interested crowd.

In April 2011 i had written an article in German on how to set up and drive SSD under Linux and never found the time to translate it. As there is still quite a few pitfalls to this endevour, Don Boyd, that is known as <dibl> on our forums, has taken to the job to translate and enhance that blogpost with his own experiences. I put it online today. Thanks much to Don for the work.

We need translators and language-maintainers for the bluewater manual. Languages in need of help are Dansk, Dutch, Czech, French, Italian, Polish, Japanese, Spanish, Russian, Romanian, Ukranian, These translations of the manual are mostly done 99%, but need to be reviewed and adopted for siduction and maintained from there on. We also need someone to coordinate the effort.

I am right now reviewing German and English, so those 2 could be used as reference for future reviewers/translators/maintainers. Technical knowhow needed besides the respective language skills is handling Git. We will do lectures in #siduction-college to make you fit for the job in no time. So, please decide to give back and put some time into this very fine manual. Right now we have only DE, EN and PT_BR online and hope, with your help, to get the others online soon.

Also, whoever feels he wants to put some time into working on the CSS of the manual, please speak up 🙂

This information was collected over the past (2010-2011) year from multiple sources, plus my experimentation on 4 different SSD models. Performance testing procedures are by devil, from his article on the German blog. I have sequentially numbered some of the paragraphs for ease of reference when discussing and/or correcting the information – corrections and improvements are welcome.

Basic Drive Math

The storage architecture of SSDs is consistent with the legacy standards for modern hard disk drives, the so-called “Cylinders-Heads-Sectors” or “CHS” geometry, and is important to understand — even if only for the next 30 minutes.

”Advanced Format” for new large hard disk drives uses 4,096 bytes per sector – we can expect to see this value used in (distant) future SSD designs

The definition of a logical “cylinder” is not fixed — in modern drives it is a theoretical construct of the number of logical heads and the number of logical tracks, however the number of logical tracks varies based on the drive size. The industry has settled upon a default configuration for most drives of 255 heads and 63 sectors per track, so the value for the number of logical cylinders on a given drive is the output of (total sectors/63)/255. For example, a 40GB OCZ Vertex2 SSD has 78,161,328 sectors, therefore it has 4865 cylinders using default heads and sectors.

SSD Alignment

Most current SSDs are using a 512KB erase block size — but verify this is the case for your SSD. This alignment guide is written for 512K erase blocks, and would not be correct for other sizes. When data are deleted or overwritten, the operation is done in blocks of 512K that are defined in the SSD firmware. For longevity of the SSD, you would prefer that each deletion or overwrite use the minimum number of needed erase blocks. (You can do the reasearch on the lifetime number of erases for each SSD memory block — it is a few thousand, more or less, depending on the technology). Partitioning the SSD with default settings, using GParted or similar tools like you do for a hard drive, will set partition boundaries (and therefore block locations within the partition) that are not in alignment with the underlying SSD firmware, resulting in two erase blocks being involved when one erase block would be sufficient for the amount of data being manipulated. Further research on the reasons for this misalignment is left to the student and his google-foo — it is all out there.

So we want to set up our SSD with partitions (and therefore block locations within the partitions) that begin at the same sector where a 512K erase block begins — that is what we mean by “alignment”.

Methods to Make Aligned Partitions

There are multiple tools and approaches to make aligned partitions. Windows 7 and later automatically aligns during installation, and there are commercial software tools for Windows users, if they want to buy it. Here we will present 3 ways that are suitable for Linux users.

Method A — “fdisk with custom cylinder definition”
NOTE 1:
fdisk is a tool in the util-linux package. Between version 2.17.2-9 (found in Debian 6 “Squeeze” Live CD, aptosid “geras” Live CD, etc.) and version 2.19.1-2 (found in aptosid “Imera” and later, and siduction 2011.1 and later, and other recent Live CDs) of the util-linux package, fdisk lost its ability to write a new CHS geometry to the partition table. You will need the version of fdisk from one of the earlier sources – probably a Hiren’s Boot CD or other legacy hard disk tool that support DOS operations will contain a compatible fdisk version.

NOTE 2:
My experiments with the current version of cfdisk indicate that it follows the current version of fdisk and forces the user to use sector 2048 for the beginning of the first partition, and refuses to allow partitioning by custom-defined cylinders.

This Method A approach offers simplicity – we set both heads and sectors to 32, resulting in cylinders that are 512KB each (32×32=1024 sectors). Thus every cylinder is on an erase block boundary, and everything is automatically aligned on the SSD. The procedure will look like this (using Debian 6 Live CD, 8GB USB stick for example):

Each “Start” sector number can be divided by 1024 exactly, and thus aligns with the 512K erase block structure of the SSD.

In the above example, I decided upon the end sector numbers based on my plan to make a 4GB OS partition, a 1GB swap space, and the balance for user data (a miniature Linux system!). Once I saw, in the first partition operation, that there were 15014 cylinders, and knowing my drive size of ~8GB, I simply divided 15014 in half, and then similarly estimated the number of cylinders needed for the swap, and then accepted the default end of the stick for the third partition. The numbers are a bit approximate – since we’re using whole cylinders it is only important to start each partition immediately after the end of the previous partition.

Method B — “fdisk with default CHS settings”

Or, we could call this method “person in a big rush, no old fdisk version handy”. But, this one requires a calculator, so better plan to exercise some patience, regardless of the rush.

We start again with an off-the-shelf USB stick for our example, “fdisk -lu” shows:

Using the newer version of fdisk found in util-linux 2.19.1-2, there is no use trying to change the heads and sector geometry because our desires will be ignored (contrary to what the man entry tells us), and we will be forced to partition by sectors anyway. When we get to the third step, where we set the beginning of the first partition, notice that we are offered only sector numbers, and the lowest beginning number offered is 2048. While this is a multiple of 512K and thus an aligned sector, we are forced to waste the preceding 1024 sectors which could be used (see the fdisk output from Method A).

Second partition (1GB swap)
Estimate a 1GB size in sectors: 15682558/8 = 1960319.75 sectors per GB
Find the nearest whole number multiplier of 1024: 1960320/1024 = 1914.375
Calculate number of sectors needed (1914×1024=1959936)
and add to the end of the preceding partition to get ending sector number: (7840767 + 1959936) = 9800703

Third partition therefore begins on sector 9800704 (evenly divisible by 1024) and ends at the end of the drive.

Method C — “gdisk”

gdisk is used to support GPT partitioning. Debian versions are found after Debian 6 “Squeeze”, for example running siduction we have:

Note that we used “p” to check the partition table before writing it, and one can see that the partitions do begin on sectors evenly divisible by 1024. It is aligned as intended.
Back in the terminal we can take a look at what fdisk sees, but in the case of the GPT partitioned drive, it cannot see the partition table:

This concludes the partitioning and partition alignment guidance. Now let’s see what tweaks we can make to maximize the performance of our system.

Optimizing SSD-based System Performance

The goal of these configuration settings, generally, is to enable TRIM, to minimize erase/write cycles that don’t add to system performance, and otherwise optimize the OS to provide a very responsive user experience.

1. Filesystem type and /etc/fstab configuration

We want to use ext4 and take advantage of the journaling feature, for data security, but we want to reduce the frequency of journal commits from the default 5 seconds to a slower rate, to extend the life of the SSD memory blocks. The “commit” mount option no longer controls the commit frequency in recent Debian distributions — instead we find the journal commit frequency is a setting in /usr/lib/pm-utils/power.d/journal-commit.

For the value “JOURNAL_COMMIT_TIME_AC=${JOURNAL_COMMIT_TIME_AC:”, I use “-120” or 2 minutes for my desktop. The default value for “JOURNAL_COMMIT_TIME_BAT=${JOURNAL_COMMIT_TIME_BAT:” is “-600”, or 10 minutes. That is way too long for my taste — I like my data, so I set it back down to “-120” also. Use the “discard” mount option to enable the SSD’s TRIM capability, and change the “relatime” option to “noatime” to eliminate unnecessary disk writes when a file is read but not changed. As a result, the /etc/fstab line that mounts the OS will look like this:

We want to mount selected filesystems as “tmpfs”, which lets the OS use memory rather than the SSD for logging and spooling. The wise user will wait for a reasonable period of time after initially installing on the SSD, before these changes are made to /etc/fstab, because until you are sure your system is stable, you should allow the logs to be written on the SSD, for later review. Logs written in memory will not survive a reboot. When you are satisfied that the system is stable and the logs can safely be lost at each reboot, add these lines to the end of /etc/fstab:

2. Outsource the browser cache to /run/shm

Since we now have the shared directory /run/shm in RAM, we can outsource the cache generated during browsing to memory, and eliminate many SSD writes. For example, in the Firefox/Iceweasel address bar we enter “about:config” and confirm the warning. Now right-click in the white space and choose “New ==> String” and we create a new entry called:

“browser.cache.disk.parent_directory”
After double-clicking the new string, we assign it the value:

“/run/shm/firefox-cache”
Now as user in the terminal create a directory:

mkdir /run/shm/firefox-cache

After a Firefox restart, browser caching happens in memory, not on the SSD.

For chromium-browser, the cache location is set with the “–disk-cache-dir=”DIRNAME” launch command option. So to outsource the chromium-browser cache:

mkdir /run/shm/chromium-cache

Open the chromium-browser launch icon for editing, change to the “Application” tab, and edit the start command to read as follows:

/usr/bin/chromium --disk-cache-dir=/run/shm/chromium-cache %U

The new browser cache directory in /run/shm will not survive a reboot. To automate this process, put the following “auto_browser_cache.sh” script in your ~.kde/Autostart folder (for KDE users), and then “chmod +x” to make it executable:

Analogous cache outsourcing configuration can be made for other browsers, if they allow the user to specify the cache location, and the startup script can be adapted to add directories for each browser that the user wants to run.

For a desktop system that remains booted for long periods, and depending on the memory capacity and browsing activities, the outsourced browsing cache could grow to a problematic size and need to be manually cleared to avoid sending the system into swapping.

3. I/O Scheduler selection

Multiple sources that you can find with a google search indicate that, for SSDs, the “deadline” and “noop” schedulers perform better than the default “cfq” scheduler. I have not done any testing to determine which is faster on my SSD installations, so that will be an exercise left to the student. Set the scheduler with an edit to the line “GRUB_CMDLINE_LINUX_DEFAULT=xxx” in /etc/default/grub, so for example:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash elevator=deadline"

4. Virtual memory settings

Depending on how much memory your system has, and how you use it, the same tweaks to vm (swappiness, vfs_cache_pressure, etc.) that you use for a hard disk drive installation can also be applied to a system installed on a SSD. Guidance is available via google search. Here are the lines added to /etc/sysctl.conf on one of my SSD installations:

Performance Testing (from Ferdinand Thommes’ article)
Before you spend time on performance testing and benchmarking your SSD, you need to determine the firmware version you have, and then check the OEM’s website and learn whether a more recent version is available. Significant performance improvements can result merely from updating your SSD firmware — follow your OEM’s instruction to install updated firmware. To check your firmware version:

From the output we copy the number immediately under “begin_LBA” and insert it in the next command:

hdparm - read-sector 1234567 /dev/sdx
# 1234567 replaced with the number from the previous command and /dev/sdx with your device ID

The output should be a longer string. Next:

rm tempfile
sync
hdparm - read-sector 1234567 /dev/sdx
# replace 1234567 and /dev/sdx with your values
< (pre>
The sectors will not be cleared instantly due to caching -- wait for some seconds. Then repeat the last command (hdparm - read-sector ...) -- it should (after a short while) come out all zeros. That means TRIM works! If you have problems with "discard" on your SSD and you have verified that your SSD does support TRIM, you can use fstrim which is in the current util-linux package (check "man fstrim"), or use the tool "DiskTrim" from http://disktrim.sourceforge.net/.

6. Throughput Benchmarking

CAUTION: You can benchmark your SSD to a premature death by subjecting it to frequent comprehensive benchmark tests!

6a. Simple hdparm test

hdparm -tT /dev/sdx

Run it twice in rapid succession — normally the second run is fastest.

Today i found the time to add all relevant info about siduction and its ressources available at this point to the Debian Derivatives Census wiki-page. pabs then added this blog to Planet Debian Derivatives Thanks for that and for all the work on Debian Derivatives.

We have had 10 days to relax and to relieve the pressure. Looking back now, on the past 6 months as well as on the result, things look quite well. At no point during that half year have i had the feeling it would not be worthwile, and i guess the rest of the team would agree to that. All in all, the work in the team was very relaxed.

First release was received well and did good on DistroWatch for a complete newcomer. So we are quite happy with the result. When it comes to bugs, there is mainly two: For one, the installer does not handle some partitions the way the user assigned them in the mount-tab of the web-interface. The other one is the handling of nonfree-firmware by fw-detect. In some cases it just adds contrib and non-free to sources.list and then, from what the user sees, it does nothing. Our best guess right now is that it times out if apt-get update takes too long on weak connections. Other than that we forgot to add some packages like policykit-1 for XFCE.

Work has been ongoing to move our website to a new place. We are waiting for Joomla 2.5 to be released. The old forum has been updated to the new look, the manual has been revised half way by now and went online yesterday. Over time we will try to improve the look and handling of the manual. I also added information about siduction to the Debian Derivatives Census today.

Starting from 15-01-2012, with the first core meeting, we will dive into the release cycle for 2012.1 and see what we want on the menu for our 2nd release. As most every successfull rockband will tell you, the 2nd one is the hardest. 🙂 You got to live up to the first one and make it even better.

Last, but not least, to make ends meet, today Dinko Sabo (vibora) returned to the team. He had disappeared 2 weeks before release and noone knew where he was. So we started worrying, after we had turned a few stones without result. As it turned out, he was in the hospital. Welcome back.

ChiliProject 2.6.0 has just been released. It includes some bugfixes for ChiliProject 2.5.0. It is suitable for use on production websites and we recommend that all users download the release as soon as possible.

We will upgrade our Installation today, so please excuse any posbible inconveniences. This should be the last planned upgrade of ChiliProject 2.x. We are planning a Upgrade to ChiliProject 3.0 after test later this month.

What’s included

2.6.0 includes 6 new features and 8 bug fixes for 2.5.0. None of the bug fixes is security related. The major highlights of this release are:

ChiliProject is now fully compatible with Ruby 1.9.3

Plugins needed by the core and user-provided plugins should now be separated. Users are advised to install their custom plugins into vendor/chiliproject_plugins from now on. This helps to distinguish plugins during updates. Existing installations with all plugins in vendor/plugins will continue to work as they used to be.

Admins using LDAP as an authentication backend can now define arbitrary LDAP filters to further narrow down the elements eligible for authentication.

rdm-mailhandler.rb which is used for receiving mails is usable again after fixing a regression introduced in 2.5.0