A running description of activity related to DragonFly BSD.

Category: Committed Code

I know OpenSSL in DragonFly was just updated, but Peter Avalos has done it again, bringing it to version 1.01e. I assume this new version is to fix some recently-exposed problems. He also has updated libdialog, which was previously not located in contrib/, as sime third-party software needed a more modern version. As a side effect from that, tzsetup in DragonFly now matches the version in FreeBSD and NetBSD. And, Sascha Wildner has updated the locale files on DragonFly, also to match FreeBSD and NetBSD.

John Marino has set gcc 4.7 as the default compiler in DragonFly. This replaces the previous default of gcc 4.4. The 4.4 version is still available, and while you can set NO_GCC44 to keep it from being built, John’s commit message notes that it’s still useful especially for some ports that don’t work with gcc 4.7.

Sepherosa Ziehau makes commits almost daily to DragonFly’s network infrastructure, but I have a hard time quantifying it into Digest posts in part because it’s often very technical. His most recent commits come with an explanation, however. He has done plenty of work to improve overall transmission speeds in DragonFly, and now he’s working on ‘fairness’. Fair, in this case, means ensuring that packet transmitting and receiving happen without either one monopolizing the connection. In real world terms, this translates to much more constant speeds. His recent commit details what he’s doing and some numbers to prove it.

Remember I said he’s improved speeds? Note that in his example, he’s reaching stable peaks of 981 Mbps. This is on a line that I assume theoretically maxes out at 1000.

Peter Avalos has updated m4 for DragonFly. This will bring us a little more in sync with the other BSDs. Also, John Marino has updated flex, which is apparently 17 years old? Meaning it hasn’t been updated in DragonFly ever, and then not in FreeBSD before that, for a long time. Looking at the timeline on the flex web page appears to match.

If you are on DragonFly 3.3, and you are running a kernel built after January 1st, there’s a bug in the way FP context is handled when the kernel supports AVX. (January 1st is when AVX support was committed.) Matthew Dillon has committed a fix and issued a note to update for everyone.

If you recall, Phoronix recently ran a bunch of benchmarks on DragonFly. One spot that didn’t look good was the “Himeno Poisson Pressure Solver”. I’m no closer to knowing what capability it actually tests other than itself, but Alex Hornung, Matt Dillon, and Venkatesh Srinivas figured out that cache coloring was the missing ingredient. DragonFly now scores the same as Linux.

I’m not sure what IFQ stands for, but Sepherosa Ziehau’s added it. It appears to be based on an idea from Luigi Rizzo called ‘netmap‘. In this case, network packets are grouped together before being placed onto the network interface’s hardware queue. That means better packet per second performance without a corresponding increase in CPU usage, as Sepherosa Ziehau’s report lists, along with needed sysctls.

If you’re running DragonFly 3.3, make sure you perform a full buildworld and buildkernel when you next upgrade. Sascha Wildner is mentioning this as a cautionary note after experiencing issues when using quickkernel, after removing a number of syscalls. Once past that point, it should be safe to go back to quickworld/quickkernel.

Matthew Dillon turned off the machdep.pmap_mmu_optimize sysctl by default, since wider testing has found some bugs. It’s only on by default on DragonFly 3.3 systems, so there’s nothing to do if you’re on 3.2-release. The feature will come back after some bugfixing.

I knew about files like /etc/services, for common IP port usages, and /usr/share/zoneinfo, for time zones, but I didn’t know that DragonFly (along with other systems) keeps a list of agreed names for various human languages defined by ISO639 in /share/misc/iso639, and it’s maintained at least in part by the Library of Congress. At least I didn’t know until Sascha Wildner updated it.

Do you use ndis(4) for a network card that would otherwise not work? Are you running DragonFly 3.3? Are you willing to run USB4BSD and see if it works? If you do, tell Sascha Wildner if his changes worked.

If you are one of the few people still wanting to read an OS/2 HPFS drive, support for it in DragonFly has been updated by Antonio Huete Jimenez. It’s read-only, but writing didn’t work well, and I’d be surprised if there’s any hpfs disks that aren’t archival, out there.

Also, Sepherosa Ziehau has updated the pktgen program to generate even more packets, even at relatively low CPU clock speeds.

The initial download of pkgsrc via Git on DragonFly is a little bit faster now, with the ‘make pkgsrc-create-shallow’ option recently added by John Marino. Note that there’s a similar option for src. It skips downloading file history.

Matthew Dillon’s putmoreofhisHammerwork into DragonFly, with notable parts being the creation of a ‘dmsg’ setup for advertising available block devices to share between machines using Hammer. To anticipate your next question: No, it’s not something you can run right now as a test; this is the underlying framework.

John Marino hasremoved gcc 4.1 from DragonFly. It was detached from the build process in 3.2, but now it’s out of there entirely. I think this affects nothing at this point other than the size of /src.

A conversation about compilers in the DragonFly base system led peeter (must) to describe his group’s use of OpenMPI on DragonFly for physics calculations. Apparently he’s had a significant performance improvement on DragonFly.

The summary result: If you’re running Postgres, you probably want to do it on DragonFly. The numbers are the best results for any BSD, even better to some extent than Linux, which has had its own issues with schedulers and Postgres. DragonFly 3.2 will include these improvements.

Matthew Dillon has created an experiment: shared page table mappings. It’s controlled by a sysctl, since it’s still experimental. The real-world effect is reducing the number of memory faults as a process uses up memory, and decreasing the overall memory usage. The obvious benchmark is Postgres speed; this makes the initial expansion of memory usage much less of an drag on speed due to a high memory fault rate.

If all this mention of faulting sounds like a problem, remember memory faults on BSD are normal; that’s how programs indicate they need more memory space by causing a fault. This is in contrast to Linux, where memory is allocated a different way. Or at least, that’s my understanding. (If you know better, please comment.)

Antonio Huete has updateddhclient(8) to match the OpenBSD version from whence it comes. I think all (most?) the BSDs use the OpenBSD dhcp client as a base now. The only user-facing change I see in a quick reading of the changes is a new ‘egress’ command line option.

Pierre Abbat noticed that bc(1)‘s usage of GNU readline something that wasn’t GNU readline made it harder to use; Sascha Wildner changed it to use libedit. Pierre’s other complaint, that BSD man page output stays on-screen when completed, is a positive feature. Linux systems that clear man page output enrage me, because I expect to be able to take advantage of my scroll buffer.

John Marino has added a ‘gcc47′ compiler ccvar, so you can build world and kernel with it. ‘It’ is actually gcc-aux, since it seems to work better than the basic (“vanilla”?) gcc47. You also get Ada support, though that wasn’t the driving reason to pick it. This is brand new so don’t try it unless you’re ready to discover issues.

Is there any other BSD able to use gcc 4.7 for world/kernel? Even 4.6? Most of the attention has been on clang.

Sascha Wildner has synced find(1) with what’s in FreeBSD, which means there’s a lot more options available – see the commit for details. Many of them are for GNU compatibility, and I’m sure I’ll forget them all. I seem to have issues remembering how to use find(1) successfully.

Sepherosa Ziehau has been doing a lotofworkonpackettransmission; far more than I link to here. The end result is startling performance on high-bandwidth links. I’m hoping for benchmarks at some point, but until then, I just wanted to publicly appreciate the work he’s done.

If you are running bleeding-edge DragonFly, libpthread was broken for a short period. If you built anything in the last … 12 hours? You may want to rebuild it. If that doesn’t describe you, it’s a nonevent.

It’s funny that I’m reporting a short-term break in bleeding-edge operating system code as any sort of surprise. It shows something about how stable DragonFly-master is most of the time.

Apparently Apache 2.4 has a bug that will cause network stalls when sending data that doesn’t line up with segment size. Sepherosa Ziehau has put in a workaround for the issue. Alternately, you can use www/apache22.

John Marino has updated libncurses, libedit, gdb, libgmp, and zlib. The release notes are helpfully contained within each commit. If that wasn’t enough, he’s also added terminfo, a future replacement for termcap, if I understand correctly.

DragonFly now has a optimizedscoreboard for SACK, thanks to Sepherosa Ziehau. What’s that mean? SACK is a way to make sure only the needed parts of a TCP transmission get retransmitted, when multiple packets are lost. The scoreboard is where the packets needing retransmission are tracked. So, the result of these improvements is better performance in packet-lossy situations.

(Please correct me if your understanding is better than mine; my explanation is based on stumbling around the Internet for a few minutes of reading.)

Sepherosa Ziehau has made changes to the initial TCP congestion window, based on a number of papers he links to in his post. The immediate effect is if you’re on DragonFly-current, you will need to do a full buildworld on your next upgrade. The long term effect could be improvements in latency by improving reactions to bufferbloat. Or not; this is pretty technical.

Peter Avalos has updated DragonF’y’s OpenSSL to version 1.0.1, in part to make future upgrades easier. See the changelog for what’s new. Look for the part specifically about 1.0.1, since the notes include unreleased material too.

If you’re trying DragonFly 3 in a virtual machine, you may have noticed some issues in booting in (for instance) Qemu. Sepherosa Ziehau committed a change that sets the sysctl hw.ioapic_enable to 0 in virtual environments. It can always be turned back on, but the recent MSI/MSI-X improvements seem to cause trouble in some virtual environment. You can also set that tunable at boot to get an initial install going.

(I haven’t had trouble in Virtualbox or VMWare, so you may or may not need this.)

Another “first BSD to try it” feature: GNU hash table support has beenadded by John Marino. These apparently speed up symbol searching during program startup, so it should improve large program startup time. Think KDE or Open Office, though I don’t have numbers to back it up.

It’s now possible to specify a jail ID when using pgrep and pkill(1), to capture processes specific to a jail. It’s similar to the same option in FreeBSD, except no compatibility issues since this option did not previously exist in DragonFly.

Here’s an interesting side effect that came up in Hammer 2 development: deleting files can potentially require modification of only one parent element. If I’m reading it right, that means deletion always takes about the same time, independent of the amount of data being deleted. Your ‘rm -rf /largedrive’ could complete, removing multiple terabytes of data before you realize it. I suppose it’s silly to complain about speedy results. Of course, being Hammer, it would still be available in history.

Thanks to John Marino’s work, it’s now possible to build the DragonFly kernel and world using gold, and have it work. You just have to set WORLD_LDVER to make it work. I don’t think there’s any user-visible change from this, other than a tiny speedup in building. I don’t know if any other BSD is using gold yet.

Alex Hornung added support for rdrand(4), the random number generator built into some Intel CPUs. That would be Ivy Bridge CPUs, which aren’t released yet, so it hasn’t been tested… but you’re covered for that day in the future when they arrive.

For the curious and technically oriented, Hammer 2 development can be watched directly by looking for any commits marked ‘hammer2′. There’s been a lot, and if you want to see the code as it flows in, here’s your chance.

ISDN support has been removed from DragonFly. It was not useful at this point, because it’s rarely used any more. It does make me feel a little sad; this was the technology everyone said was the future before cable modems and DSL were figured out.

John Marino has added support for RELRO in DragonFly, which makes it the first BSD to have it. That’s great news! What is it? Apparently a guard against memory corruption or overflow in the linker. His commit message gives better details.

Matthias Schmidt found a discussion about DragonFly’s password encryption. The result, if I am reading it correctly, is that brute-forcing the password from available hashes is quicker than it should be. Matthias also found a contributed fix. Samuel Greear updated to match the reference SHA implementation also in Linux, with this very pertinent warning.

Matthew Dillon has a very detailed commit message with changes to make sure Hammer will run overnight cleanups in situations as low as 256M of RAM. I think you can find that much RAM in breakfast cereal boxes these days.