We won't be including any such tool until it's stable and integrated into e2fsprogs upstream. It would be far, far worse to ship a broken version of such a tool than to ship no defragmentation utility at all. (Besides, our kernel doesn't have the online defragmentation patch yet; I gather this is quite recent work.)

While I'm going to leave this bug open since otherwise I imagine we'll just get another copy of it, please do not make further requests here; this is entirely the wrong place for this kind of request. The work needs to happen upstream and *then* be integrated into Ubuntu.

The online defrag patch isn't ready for prime-time yet. It's getting much better, but it's still not at the point where I'm comfortable merging it into the mainline kernel yet, and the userspace API's are still not yet set into stone. So the userspace component is not going into e2fsprogs upstream until I'm satisfied the kernel piece is ready for mainline. The current status is that I gave the ext4 defrag developers some comments, and I haven't heard back from them in a few weeks. Probably time for me to ping them again....

Reading the many discussions/forums etc. concerning ext4, I believe one of the MAIN reasons that folk are looking to get a defrag tool running is to 'refresh' files on their disk having moved from ext3 to ext4, rather than worrying about the (minimal) benefits of a defrag itself. Indeed, that is why I came across this thread because I upgraded from Ubuntu 8.10 to 9.04, switched to ext4 and then wanted to refresh all of my files to the new format - apparently the most effective way to do this is using a defrag tool.

Maybe if there was an alternate (and 'safe/easy') way to refresh all files on the disk to ext4 most folk would be happy and maybe wait for 9.10KK to get a completely stable defrag installation?

I don't think there is any simpler way to do this than to use the defrag itself. I'm ok with waiting until it's been tested enough to become part of e2fsprogs which likely won't be around until 9.10. That's ok though.

I know this is a big wishlist item, but I doubt it'll become part of 9.04 since it's been released and that will be a big testing item.

The defragmentation support in ext4 is still a work in progress. The current e4defrag essentially creates a new file that is the same size is the existing file, checks to see if it is less fragmented, and if so, invokes a ioctl which atomically substitutes the blocks from the new file to the new file, while preserving the contents of the file.

It however, doesn't have any concept trying to make the file system better from a global point of view, in terms of (a) making sure the free space is not fragmented so as to avoid the file system becoming fragmented again in the near future, and (b) using a global file system wide strategy to move files aside to "make room" for a large fragmented file. It also hasn't undergone enough testing that it's really a good idea to recommend its use in production settings.

There are kernel patches that will provide more functionality to enable a more intelligent userspace program, but (a) those patches still need to be evaluated, so they are not yet in mainline, (b) the enhancements to improve e4defrag to take advange of these new ioctls have not yet been written, and that work is non-trivial, and (c) there is still work needed to improve ext4's base allocation algorithms to enable the current e4defrag to work more efficiently and to prevent fragmentation of the free space to avoid fragmentation problems on full filesystems in the first place.

I've been working on (c), and there are patches in the ext4 patch queue that address the last point. But quite frankly, e4defrag work is somewhat lower priority compared to another things that that we have been working on as of late, such as, integrating the 64-bit block number support into e2fsrpogs' mainline, fixing direct I/O into preallocated blocks, and so on. Some of these items are less important to Ubuntu, as a desktop distribution but they are important to the enterprise server companies and distributions that have been donating engineering effort to ext4. If Canonical is interested in pushing ext4 defragmentation support and contributing some real kernel and userspace engineering support to accelerate this project, please contact me directly. Otherwise it will keep chugging along, but it will be a while before it is really something I would recommend for end-user use.

Thanks Ted. Even if it's not ready for heavy production use, it sounds reasonable to have it included so adventurous users can test it out. Perhaps that will catalyze work on tools to visualise the level of fragmentation, and in turn tools to improve the defrag.

As long as adventurous users realize that e4defrag might destroy their data, sure. There has been some discussion about initializing requiring e4defrag to check for the existence of an environment variable, "I_KNOW_E4DEFRAG_MAY_DESTORY_MY_DATA_AND_WILL_DO_BACKUPS_FIRST" before it will run.

One of the big differences between filesystem utilities and other programs is that failures with say, a cr*p proprietary video driver that causes a system crash whenever you exit a game is after you reboot, little harm is done. But if e4defrag ends up destroying data, users tend to get awfully cranky.

Eric Sandeen from Red Hat has contributed code to a test suite to provide very basic defragmentation testing, but making sure e4defrag does the right thing under all circumstances (i.e., sparse files, files with preallocation space, files which I/O is taking place at the same time as the defragmentation, etc.) still needs to be added.

As far as tools to visualize the level of fragmentation, Andreas Dilger from Sun Microsystems has already donated a tool to help visualize the level of free-space fragmentation, e2freefrag, which is already in the git mainline sources.

One thing comes to my mind with respect to people trying this out. I recall reading that e4defrag was going be the proposed way to convert files to using extents after migrating from ext3. But I get "Operation not supported" when I try to defrag files older than the date of migration, so it looks like e4defrag doesn't work on non-extents files yet (that or something strange is going on).

Testers should be aware of this so they won't expect to be able to migrate to ext4 then immediately run e4defrag and have it work. If you migrated from ext3, the older files won't defrag.

You can convert an individual file to use extents by using "chattr +e file1 [...]"; in theory the ioctl used by chattr +e could be added to e4defrags. This functionality was added to chattr after e2fsprogs 1.41.6. The kernel code to support the ioctl has been in the kernel for a long time, but it hasn't gotten much testing. The same thing applies in terms of potential for data loss. It's been lightly tested, but hasn't received a major shakedown, mainly because it hasn't been a high priority. If someone would be willing to do some heavy testing of what happens if the file is sparse, or is being written to at the time, etc., for both chattr +e and e4defrag, or better yet could write some test code which could be added to a regression test suite, please contact the linux-ext4 mailing list at vger.kernel.org. The test effort would be greatly appreciated.

This feature request is now almost one year old.
xfs support extents and defrag since long time, but it doesn't support shrink operations.
ext4 support shrink and extents, but doesn't have defrag.
What will come first? Defrag on EXT4 or shrink on XFS?

From a user perspective, unfortunately, we don't need 10 filesystems doing each one, one different things. We need one filesystem doing all these 10 things.

@Theodore: why you wrote "crap *proprietary* video driver"? Frankly, but really frankly, as long as I have seen "crap video drivers" have been usually opensource. Even the Intel - which is not reverse-engineered - is crap compared to nvidia driver and nvidia tools.

E2defrag should hopefully be releasable in Karmic+1. There are still a lot of bugs that are still being fixed in the defrag code, some of which could cause data loss. They work fine if the system isn't under stress, sure, but acid test is to make sure things work OK even when the system is under memory pressure and swapping heavily, or when the file is being actively modified at the point where the defrag takes place. The kernel patches are there, and e2defrag is in the upstream development tree (which hasn't been released yet). So it will happen, but it's always been a matter of time.

The Intel driver has never crashed on me. The Nvidia driver was crashing regularly --- users were reporting regular crashes while using it during the Januty development cycle. Unfortunately, many Ubuntu users are Windows refugees, so they see nothing wrong with the system locking up whenever they exited a 3D game (I think it was World of Goo in one case, if memory serves correctly). That is simply totally unacceptable.

I'm old school, which means job #1 for file systems and video drivers is that they don't lose data and they don't cause your system to crash. It's not about "number of file system features", or "hundreds of FPS when playing game X".

I've recently been looking into this a little more. The kernel support is there. The development is there. It sounds like it's still planned to be available in 10.04.

I did some testing and reading. A migration to a fresh Ext4 partition seems to yield better performance than the e4defrag. Of course ymmv.

If you want defrag for performance, it may be a good idea to check and see if your file system that would benefit from this.
sudo e2fsck -nfv /dev/sdax
The worst I had was 1.1% file fragmentation and 0.6% on the same 100MB partition. Considering this is where I drop all my custom kernels and do a lot of altering here, that's pretty darned low. I wouldn't even consider wasting my time with a defrag of ext4 unless I'm running over 3.0% which is still pretty dang low.

I am excited to see how this and the other features progress. Ext4 has been awesome when it comes to my servers.

Just performed a check on my 1TB partition (I use it in my Gentoo home server).
The partition is one year and one month old
917GB total, 871GB used, 37GB free, 10GB reserved
it is used as storage for audio and video files and for rtorrent.

do it.... please :-)
This issue is still there from one year and I would like to migrate my XFS filesystem.
This feature was added to Veritas filesystem, during the pleistocene epoch, and a little bit later, after the the firt glaciation, was seen first time in XFS.

> I've recently been looking into this a little more. The kernel support
> is there. The development is there. It sounds like it's still planned to
> be available in 10.04.
>
There is no plan for this to be made available until e2fsprogs upstream
feels sufficiently happy about the tool to include it as standard in
their releases.

When I take a look at this thread, I see a lot of ppl expecting too much of a defragmentation tool. Please keep in mind that almost all linux file systems will try to avoid fragmentation by themselves.

@bmhm: I am not pretty sure if you're right.
First, I would state that this linux is not a kind of thing where everything is magic, and so fa and so on, and file self-manage fragmentation and filesystem never fragment.
Veritas and XFS, have a defrag tool from a long time.
As far fragmentation is concerned, if I am not wrong, all journaling filesystems have fragmentation issues. And Linux is not different.
What you're saying (again: if I am not wrong) was true for paging filesystems, where each file had an offset to the end. Furthermore, a paging filesystem almost full, was fragmenting as well (because all offset were going to be filled).
I suspect the structure of modern filesystems is completely different and we have fragmentation issue.

I recently migrated from XFS to ext4 and the FS is pretty good.
What I am missing is:
1. such tool to defragment FS
2. xfsdump, which was a very good tool for backups.

Cool, the e4defrag built with those patches seems to work fine. The check function (-c) on my root partition reported a fragmentation score of zero (I guess it must be an average) but did find a number of fragmented files, eg

If I run it on a directory, it reports a number of 'successes' and 'failures'. I suspect the failures are for files that are locked (wpa_supplicant was one such file), ie not all files can be defragged online.

The "relevant files defrag" always reports a 100% failure rate for some reason.

Build/install instructions are in the INSTALL file. Note that e4defrag is built in the misc folder but doesn't install with the rest when you do a sudo make install.

There are also patches for the relevant file defragmentation mentioned in the slides (comment #32), ie at http://marc.info/?l=linux-ext4&m=128272678710600&w=2. The first three are for the kernel (it looks like they are based on 2.6.34 and didn't cleanly apply to my 2.6.37-rc1 kernel so I didn't apply them, which is likely why -r kept failing for me) and the last two are for e2fsprogs. I guess you don't need them if you are just trying out the file/directory defrag. From the slides, it looks like file/dir defrag just allocates a contiguous space for the defrag file and moves it into this space.

BTW, I don't recommend running sudo make install on e2fsprogs. When I attempted to reboot, the system didn't recognise the new fsck and hence would only mount root in read-only mode. (Oddly enough, fsck still ran fine.) I had to boot from a live CD, chroot to my broken partition, and reinstall e2fsprogs and util-linux using apt-get. e4defrag still runs fine by itself.

Shake is a defragmenter that runs in userspace, without the need of patching the kernel and while the system is used (for now, on GNU/Linux only).
There is nothing magic in that : it just works by rewriting fragmented files. But it has some heuristics that could make it more efficient than other tools, including defrag (see ck.kolivas.org/apps/defrag/) and, maybe, xfs_fsr.

Does this mean that as usual, we won't get this for the last stable release (currently oneiric) nor for the last LTS (currently Lucid)? Will it get backported or we really must rush with the OS upgrade if we want it?

Oh, and concerning ext4 fragmentation, I find it strange that so much fragmentation as cited above... The most fragmented ext4 FS I have is about 8%, and strangely, another one which is almost full and its content moderately often changed, stays low at about 2% even though I've expected much more.

I believe such a low degree of fragmentation I have is due to Extents, and I have specifically enabled them for my ext4 filesystems. Perhaps in those cases with higher ext4 fragmentation extents are turned off? I'm not sure are they enabled by default, but I believe they are disabled if the FS is upgraded from ext3, so I'd suggest anyone to manually enable it (in fstab, perhaps also with tune2fs); I'd recommend rebooting to single mode immediately after that, and then manually fsck all filesystems for which extents were turned on. And remember, backup often ;)

Also, when extents are turned on, perhaps running the Defrag script or Shake would help lower the fragmentation...