Bug Description

- Restore fsync during package unpack (LP: #559915). This is now done
by deferring the fsync and rename for normal files in tar extraction
so that it's done in one pass afterwards, to avoid massive I/O
degradation due to the serialization from each write + fsync. When
creating hard links to normal files on extraction use the .dpkg-new
filename for source as the file is not yet in place due to the rename
deferral.

I just installed the same hardware, from the same USB stick, in an identical configuration, one on ext4 and the other on ext3.

On ext4, this took 19 minutes, 20 seconds. On ext3, this took 9 minutes, 8 seconds.

This is a 100% performance hit on Server installs. It takes now takes over twice as long to install Ubuntu servers.

While I can understand that the fsyncs() are necessary for power loss situations on apt-get upgrade/dpkg operations *after* the system has been installed, it should not be necessary at Server install time. If you lose power during a d-i installation, you will clearly need to start from scratch anyway.

The desktop installer does not suffer from this since ubiquity installations simply transfer the live image.

DEVELOPMENT BRANCH: Addressed in a merge from Debian by using one large sync before embarking on renames, rather than lots of little fsyncs.

TEST CASE: Install Ubuntu Server; measure time taken in "Select and install software" step. Compare 10.04 as released to the 10.04.1 candidate CD images which should be available for testing in the near future.

The default file system for installations of Ubuntu 10.04 LTS is `ext4`, the latest version in the popular series of Linux extended file systems. `ext4` includes a number of performance tuning changes relative to previous versions such as `ext3`, the file system used by default up to Ubuntu 9.04. These generally produce improvements, but some particular workloads are known to be significantly slower when using `ext4` than when using `ext3`. If you have performance-sensitive applications, we recommend that you run benchmarks using multiple file systems in your environment and select the most appropriate.

In particular, the `dpkg` package manager is known to run significantly slower on `ext4` (causing installations using the server or alternate install CD to take on the order of twice as long as before). `ext4` does not guarantee atomic renames of new files over existing files in the event of a power failure shortly after the rename, and so `dpkg` needs to force the contents of the new file out to disk before renaming it in order to avoid leaving corrupt zero-length files after power failures. This operation involves waiting for the disk significantly more than it strictly needs to, and so degrades performance. If fast package management operations are most important to you, then you should use `ext3` instead. (Bug:570805)

The simplest way to select a different file system such as `ext3` at installation time is to add the `partman/default_filesystem=ext3` boot parameter when starting the installer. If you are deploying Ubuntu automatically using Kickstart or preseeding, then you can set a different file system in the partitioning recipe instead.

But it looks like either the release notes or the ext4 documenation needs updating:

===
Release notes
===
`ext4` does not guarantee atomic renames of
new files over existing files in the event of a
power failure shortly after the rename

===
ext4 docs ( http://www.mjmwired.net/kernel/Documentation/filesystems/ext4.txt#338 )
===
the data blocks of the new file are forced
to disk before the rename() operation is
committed. This provides roughly the same level
of guarantees as ext3, and avoids the
"zero-length" problem that can happen when a
system crashes before the delayed allocation
blocks are forced to disk.

The explanation in the release notes is confusing. First of all, as Jakob wrote above, ext4 starting with 2.6.30 provides similar behavior to ext3 with data=ordered. So, it is not clear why fsync() is necessary on ext4, but not on ext3.

More importantly, the default mode for ext3 was changed to data=writeback in 2.6.30, which does not provide the same guarantee as "ordered". Even if the default was changed to be "ordered" in Ubuntu kernel, users still may use an upstream kernel or change the mode to "writeback" on ext3, because the "ordered" mode has horrible latency (20 times or more than "writeback"). Does it mean that anyone running ext3 with data=writeback can face corruption of their package repository?

I wonder if this says what you meant to say ( from the kernel.org bug ):
---
auto_da_alloc doesn't detect the replace-via-rename (at least in the
case of dpkg.)
---
Is dpkg really ***replacing***-via-rename when installing new packages?
Where do the old files that are overwritten come from?

This is probably easier to test if you're seeing problems with dpkg performance after installation, although I do plan to hack it into an installation environment after I get back from UDS and do some comparisons. Please report before-and-after timings if you can, preferably with a cold cache.

If this makes a significant improvement, I'll get it into Ubuntu 10.04.1.

dpkg from Colin Watsons PPA solved the problem: I installed "Blender" and the unpacking phase of the 10Mb package took less than a second. And I experience no more rattling disks when installing packages.

Without it, a power-failed install may have zero-length or wrong-content files.

With it, a power-failed install still has a broken package -- each individual file may be fully there or fully not-there, but there will be missing files.

It doesn't help achieve a successful package install. In fact, it _increases_ vulnerability to power failures by making the vulnerable time window more than twice as long (all of the added time is vulnerable time, while some of the original time must be safe prep time).

Either way, system powers back on with a broken package. Either way, the user or the dpkg system must deal with it.

dpkg _should_ do a regular sync() after each package; I imagine (without checking source) that it already does. It already has notes on which packages were in transition. Make sure _those_ are fully sync'd, fsync'd if that's the right way to do it -- those tell dpkg where to pick up, which package to fix, after the power cycle.

I think this code should be retracted, even for normal post-install package installs. Cleanup is going to be needed after a mid-install power failure either way; don't make users suffer through slow, noisy, HD-punishing package installs for no [or negative] benefit.

Accepted dpkg into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!