Matthew Dillon wrote a roundup post summarizing all the changes he’s made to DragonFly to improve SMP performance in the last few weeks. He’s removed almost all contention from DragonFly. This means better performance, scaling upward depending on the number of processors.
‘monster’, the system that builds all 20,000 items in dports, can complete the run in 15 hours. Compare this to the 2 weeks it used to take me to build the 12,000 packages in pkgsrc. This is admittedly on different hardware and different packaging systems, but it gives a sense of the scale of the improvement.
The ‘poweroff’ command, the equivalent of ‘halt -p’, has been added based on a suggestion from Robin Hahling.
Matthew Dillon was using poudriere, the dports build tool, on a 48-core system. Poudriere was building all 20,000+ dports, so the machine was quite busy. He decided to get rid of as much contention as possible, and he’s listed all the ways DragonFly’s been streamlined by these efforts. We need to revisit some of our previous benchmarks…
If you’re watching for a certain event, dmesg(8) on DragonFly now has a -f flag that will display new output as it’s logged, sort of like ‘tail -f’.
While looking for a different bug, Matthew Dillon made some changes in the way memory usage is totaled in DragonFly. You’ll see this most when using ‘systat -vm 1′ or ‘vmstat’, probably.
John Marino has accomplished the major task of updating gdb/kgdb, to version 7.6.1 for DragonFly.
John Marino has put in a large patch to DragonFly 3.5, updating all sorts of language-related items. As he warns, you will need a full buildworld/buildkernel in a specific order to update. On the plus side, you can now probably use your native language for nvi and for git.
Matthew Dillon’s committed the work by Daniel Flores on Hammer 2 compression and Mihai Carabas’s vkernel hardware support - both Summer of Code projects. There’s a good amount of detail in the commit messages describing the work and what it changed; I expect more Summer of Code work to be getting committed…
Note: you’ll want to do a full update.
It’s now possible to use systat(1) to see per-connection speeds and pftop status, thanks to Matthew Dillon.
Antonio Huete Jimenez has committed his work on “dirfs”, a filesystem that lets you mount directories from your host machine within the running vkernel environment. It’s a sort of shared folders for vkernels. See the commit message for usage details.
Francois Tigeot has ported TTM to DragonFly from FreeBSD and I think a bit from OpenBSD. All this work has led to an update in the driver porting notes.
killall -T will now kill all processes associated with the current tty, except parents of the killall process itself. It’s a shortcut to “kill all these runaway items I started by accident”.
Apparently Sepherosa Ziehau has been improving DragonFly’s route table performance under extremely heavy load. (e.g. run efficiently; don’t die) I don’t have a definitive commit message to point at, but looking at his recent commits are a good start.
Thanks to Sepherosa Ziehau, Netperf will now calculate Jain’s fairness index. That is a formula that is both interesting and unfamiliar to me. Not that I understand it, of course – it’s just because it has a neat name.
Because Sepherosa Ziehau changed mbstat, anyone on bleeding-edge DragonFly will need to rebuild world, or else netstat will become confused.
There’s support for a new timer mechanism in DragonFly 3.5, for x86_64 users: TSC. Sepherosa Ziehau added support and has described how to disable it - it’s on by default. It speeds up some very basic (and frequently used) system calls.
Sepherosa Ziehau has added a sort of queuing to altq, where TCP ACKs get higher priority. You may have seen this in any number of pf configurations, where returning data is given its own queue to keep high-volume transfers from slowing themselves down because the acknowledgements can’t get back to the sender. His commit has statistics on the performance improvement. He also added a ‘netrate‘ tool for calculating results from using netperf.
If you’re using DragonFly 3.5, your next update should be a full buildworld. That’s because John Marino is adding the framework for symbol versioning. This means that individual library (.so) files will internally keep track of newer and older symbols. The current behavior is to name the files differently, which can cause problems if an expected, linked file is missing – even if the needed symbols are present. The basic framework is being added now, and will be turned on all at once, to minimize the number of times that full buildworld is needed.
The ‘amd64′ specific parts of kernel architecture have been removed, since x86_64 covers all that. As a side effect of other changes, John Marino warns that upgrading DragonFly from a version older than 3.4, to a version newer than 3.4, will require an intermediate step of going to 3.4 first. e.g. If your machine is a DragonFly 3.0 system, you will need to upgrade to 3.4 before moving to, say, 3.6 once it is out. This won’t matter for some months, since the next release is months off.
SO_REUSEPORT has been added and turned on by Sepherosa Ziehau. This is an implementation of a feature found in the Linux kernel. Check the very lengthy commit message for a description of what it does.
This time it’s less and libedit, updated by John Marino.
John Marino has committed updates for libmpfr, diff utils, grep, and libexpat/libbsdxml. Libmpfr, the one item that I suspect doesn’t spring instantly to mind, is a library for floating-point computation.
Peter Avalos has updated ftp in DragonFly. It’s actually tnftp, which is the same base ftp client used in FreeBSD/NetBSD/Mac OS X/etc. It’s the 20121224 version, and the 3.4 release branch has it too.
OpenBSD has a new identd daemon. Is identd used for anything other than verification when connecting to an IRC network? I’ve never seen it in another context.
Peter Avalos has committed another batch of updates to sh(1), from FreeBSD. I was going to comment on how strange it was to see software getting updated so many years later; you’d think everything there was to update for /bin/sh had been done at this point. Digging casually, the oldest bit on sh that I can find is from 1991 – 22 years old. The man page mentions a rewrite in 1989 based on System V Release 4 UNIX, and there were versions of sh all the way back to version 1.
Here’s a trivia question – what’s the oldest Unix utility, and what’s the oldest code still in use? I don’t know the answer.
Here’s 3 recent and different commits to DragonFly that I’m commenting on all at once:
- Peter Avalos upgraded libarchive in DragonFly to 3.1.2, with a note of the changes. An ordinary and appreciated update.
- Sascha Wildner updated the ISO639 file to include the newest update: “Standard Moroccan Tamazight”. There’s no particular utility to that; I just like saying “Standard Moroccan Tamazight” out loud.
- Work on poudriere, the utility for bulk-building DPorts packages, has caused some nice speedups for DragonFly in extremely stressful situations. See one of Matthew Dillon’s recent commits.
I really wish the other BSD projects would include commit lines in the mail message subjects, so it was easier to catch things like these.
The new vm.read_shortcut option has been turned on by default by Matthew Dillon, which should lead to some performance improvements. That improvement has been measured for tmpfs, at least. There’s also some buffer cache improvments that help on x86_64 systems, too.
Update: As Venkatesh Srinivas pointed out, tmpfs also no longer uses the mplock, so it’ll take better advantage of multiple processors.
John Marino proposed a method for backing up world when upgrading, for those rare but catastrophic cases where the installed programs can’t run. After some discussion, he committed an automatic backup method, and there’s a ‘restoreworld’ target to take advantage of it.
The kernel already gets renamed to kernel.old as a backup, if I remember correctly.
As Sepherosa Ziehau mentions in his latest commit, DragonFly now collects IFNET/IFADDR statistics on a per-CPU basis. This makes it more accurate, but may mess with any third-party program that accessed it directly. I don’t know if there’s anything in pkgsrc that does that…
I know OpenSSL in DragonFly was just updated, but Peter Avalos has done it again, bringing it to version 1.01e. I assume this new version is to fix some recently-exposed problems. He also has updated libdialog, which was previously not located in contrib/, as sime third-party software needed a more modern version. As a side effect from that, tzsetup in DragonFly now matches the version in FreeBSD and NetBSD. And, Sascha Wildner has updated the locale files on DragonFly, also to match FreeBSD and NetBSD.
Added: Peter Avalos has updated OpenSSL to version 1.0.1d – see the changelog.
Removed: support for ISA sound cards, by Sascha Wildner. Goodbye sb16; I’ll remember you fondly.
The emx(4) driver now has support for multiple TX queues, but it’s not on by default. There’s scenarios where multiple queues work out with that hardware, but you have to be sure you are actually in the right setup for that first. Check Sepherosa Ziehau’s commit message for the details.
John Marino has set gcc 4.7 as the default compiler in DragonFly. This replaces the previous default of gcc 4.4. The 4.4 version is still available, and while you can set NO_GCC44 to keep it from being built, John’s commit message notes that it’s still useful especially for some ports that don’t work with gcc 4.7.
GCC version 4.7 is already available now in DragonFly 3.2, but it’s not the default compiler. John Marino intends to make it default for the next release. What’s that mean for us? Nothing other than a new compiler, since he’s already fixing related issues.
Sepherosa Ziehau makes commits almost daily to DragonFly’s network infrastructure, but I have a hard time quantifying it into Digest posts in part because it’s often very technical. His most recent commits come with an explanation, however. He has done plenty of work to improve overall transmission speeds in DragonFly, and now he’s working on ‘fairness’. Fair, in this case, means ensuring that packet transmitting and receiving happen without either one monopolizing the connection. In real world terms, this translates to much more constant speeds. His recent commit details what he’s doing and some numbers to prove it.
Remember I said he’s improved speeds? Note that in his example, he’s reaching stable peaks of 981 Mbps. This is on a line that I assume theoretically maxes out at 1000.
Peter Avalos has updated m4 for DragonFly. This will bring us a little more in sync with the other BSDs. Also, John Marino has updated flex, which is apparently 17 years old? Meaning it hasn’t been updated in DragonFly ever, and then not in FreeBSD before that, for a long time. Looking at the timeline on the flex web page appears to match.
Sepherosa Ziehau has added a generic form of support for multiple transmit queues in DragonFly. This means less contention when transmitting. It’s not done; he has drivers to set up and as he said, it’s “step 1 of many“.
If you are on DragonFly 3.3, and you are running a kernel built after January 1st, there’s a bug in the way FP context is handled when the kernel supports AVX. (January 1st is when AVX support was committed.) Matthew Dillon has committed a fix and issued a note to update for everyone.
If you recall, Phoronix recently ran a bunch of benchmarks on DragonFly. One spot that didn’t look good was the “Himeno Poisson Pressure Solver”. I’m no closer to knowing what capability it actually tests other than itself, but Alex Hornung, Matt Dillon, and Venkatesh Srinivas figured out that cache coloring was the missing ingredient. DragonFly now scores the same as Linux.
Tangentially related, this cache coloring is happening in nmalloc, which is now used on 64-bit DragonFly systems. The previous one, dmalloc, had problems in long-running programs.
I’m not sure what IFQ stands for, but Sepherosa Ziehau’s added it. It appears to be based on an idea from Luigi Rizzo called ‘netmap‘. In this case, network packets are grouped together before being placed onto the network interface’s hardware queue. That means better packet per second performance without a corresponding increase in CPU usage, as Sepherosa Ziehau’s report lists, along with needed sysctls.
If you’re running DragonFly 3.3, make sure you perform a full buildworld and buildkernel when you next upgrade. Sascha Wildner is mentioning this as a cautionary note after experiencing issues when using quickkernel, after removing a number of syscalls. Once past that point, it should be safe to go back to quickworld/quickkernel.
Matthew Dillon turned off the machdep.pmap_mmu_optimize sysctl by default, since wider testing has found some bugs. It’s only on by default on DragonFly 3.3 systems, so there’s nothing to do if you’re on 3.2-release. The feature will come back after some bugfixing.
There’s been a few changes to reduce memory usage; this may not affect you unless you have an extremely busy machine, but it won’t hurt.
I knew about files like /etc/services, for common IP port usages, and /usr/share/zoneinfo, for time zones, but I didn’t know that DragonFly (along with other systems) keeps a list of agreed names for various human languages defined by ISO639 in /share/misc/iso639, and it’s maintained at least in part by the Library of Congress. At least I didn’t know until Sascha Wildner updated it.
Updated: Birthstones and flowers. Don’t know why.
Matthew Dillon’s committed a bunch more Hammer2 work. No, it’s not usable yet. Look at the commit messages for details on how he’s setting up multi-master volume information, though.
Do you use ndis(4) for a network card that would otherwise not work? Are you running DragonFly 3.3? Are you willing to run USB4BSD and see if it works? If you do, tell Sascha Wildner if his changes worked.
If you are one of the few people still wanting to read an OS/2 HPFS drive, support for it in DragonFly has been updated by Antonio Huete Jimenez. It’s read-only, but writing didn’t work well, and I’d be surprised if there’s any hpfs disks that aren’t archival, out there.
Also, Sepherosa Ziehau has updated the pktgen program to generate even more packets, even at relatively low CPU clock speeds.
The initial download of pkgsrc via Git on DragonFly is a little bit faster now, with the ‘make pkgsrc-create-shallow’ option recently added by John Marino. Note that there’s a similar option for src. It skips downloading file history.
Peter Avalos has updated OpenSSH in DragonFly to 6.1p1. This looks to be a bugfix release, but check the changelog for details.
John Marino’s committed bmake as the replacement to make, as mentioned previously. You should probably do a full buildworld/kernel sequence. This of course only affects you if you are on DragonFly 3.3.
Matthew Dillon’s put more of his Hammer work into DragonFly, with notable parts being the creation of a ‘dmsg’ setup for advertising available block devices to share between machines using Hammer. To anticipate your next question: No, it’s not something you can run right now as a test; this is the underlying framework.
John Marino has removed gcc 4.1 from DragonFly. It was detached from the build process in 3.2, but now it’s out of there entirely. I think this affects nothing at this point other than the size of /src.
The kernel in DragonFly is now SMP by default. The “SMP” option in the kernel config is no longer needed, so it’s been turned into a no-op. You don’t have to update your custom kernel config… yet.
A conversation about compilers in the DragonFly base system led peeter (must) to describe his group’s use of OpenMPI on DragonFly for physics calculations. Apparently he’s had a significant performance improvement on DragonFly.
Along similar lines, John Marino helped out by bringing in libssp and libgomp for gcc 4.7 for use with OpenMP. (This is in DragonFly 3.3, not 3.2).
Remember the new scheduler work? Well, it continued, and now Francois Tigeot has posted pgbench benchmarks of the progress and benchmarks of DragonFly vs. other operating systems. The links are to PDFs; scroll down as each have multiple pages.
The summary result: If you’re running Postgres, you probably want to do it on DragonFly. The numbers are the best results for any BSD, even better to some extent than Linux, which has had its own issues with schedulers and Postgres. DragonFly 3.2 will include these improvements.
John Marino has accomplished the difficult task of putting gcc 4.7 into DragonFly. Version 4.4 is still the default, and the older 4.1 version has been disabled. If you want to try this newer version, setting WORLD_CCVER=gcc47 will build kernel and world that way too. If you’re curious about what’s different in this version of gcc, there’s a 4.7 changelog.
Are we the only BSD with this new a version in base? I think so.
P.S.: You’ll want to do a full buildworld if you’re running DragonFly 3.1
P.P.S.: you may need to put ‘NO_GCC47=true’ in make.conf, going from IRC comments.
P.P.P.S.: Nope, now it’s fine.
The combination of Mihai Carabas’s successful Summer of Code work on the scheduler and the recent Postgres benchmarking got Matthew Dillon to start thinking about making UNIX domain sockets work better, a shortcut around the buffer cache, scheduler improvements and then a new default scheduler, along with a change in idle CPU behavior. The best place to understand all the changes is in his long post to users@.
We should have benchmarks soon to show the performance improvements from all this.
If you do, they don’t get cleaned up during the normal ‘hammer cleanup’ nightly routine. Chris Turner has added a way to manually specify them as a cleanup target.
I’m pretty sure in this case ‘offline’ means ‘nothing streaming to it from a master disk’. I think.
Matthew Dillon has created an experiment: shared page table mappings. It’s controlled by a sysctl, since it’s still experimental. The real-world effect is reducing the number of memory faults as a process uses up memory, and decreasing the overall memory usage. The obvious benchmark is Postgres speed; this makes the initial expansion of memory usage much less of an drag on speed due to a high memory fault rate.
If all this mention of faulting sounds like a problem, remember memory faults on BSD are normal; that’s how programs indicate they need more memory space by causing a fault. This is in contrast to Linux, where memory is allocated a different way. Or at least, that’s my understanding. (If you know better, please comment.)
These are small, but they make life easier: Hammer now has a scoreboard file, for viewing of mirror-streams running in the background. There’s also a ssh-remote directive, so you can use ssh without enabling an interactive shell, and a HAMMER_RSH environment variable so different remote shells can be used. These are all for Hammer 1.
Antonio Huete has updated dhclient(8) to match the OpenBSD version from whence it comes. I think all (most?) the BSDs use the OpenBSD dhcp client as a base now. The only user-facing change I see in a quick reading of the changes is a new ‘egress’ command line option.
John Marino is working on updating tcl in pkgsrc. It’s apparently quite messy to update, which may be why it has sat out of date for some time. Never one to rest, he’s also been making FUSE filesystems work on DragonFly. (Here’s a FUSE explanation, if you need it.)
Also this. Someday I’m going to write a “games on DragonFly” feature, or series.
Matthew Dillon recent posted a status report for Hammer 2. Of interest is the spanning tree protocol being built to handle messages between Hammer volumes. As he says in the message:
For example, we want to be able to have millions of diskless or cache-only clients be able to connect into a cluster and have it actually work…
(No, it doesn’t do this, yet.)
Pierre Abbat noticed that bc(1)‘s usage of
GNU readline something that wasn’t GNU readline made it harder to use; Sascha Wildner changed it to use libedit. Pierre’s other complaint, that BSD man page output stays on-screen when completed, is a positive feature. Linux systems that clear man page output enrage me, because I expect to be able to take advantage of my scroll buffer.
Peter Avalos has updated a bunch of third-party software: tcpdump, libpcap, libarchive, tnftp, xz, and OpenPAM. Thanks Peter! If you need more info on what these things are, the information is out there.
John Marino has added a ‘gcc47′ compiler ccvar, so you can build world and kernel with it. ’It’ is actually gcc-aux, since it seems to work better than the basic (“vanilla”?) gcc47. You also get Ada support, though that wasn’t the driving reason to pick it. This is brand new so don’t try it unless you’re ready to discover issues.
Is there any other BSD able to use gcc 4.7 for world/kernel? Even 4.6? Most of the attention has been on clang.
Nuno Antunes is still working on that netgraph upgrade. Among other changes, ng_tty has been added. What’s it do? Something with ppp, I think.
Based on a suggestion from Venkatesh Srinivas, tmpfs now uses a red-black tree for directory lookups, and is also now faster. Credit goes to Johannes Hofmann for doing the testing.
Sascha Wildner has synced find(1) with what’s in FreeBSD, which means there’s a lot more options available – see the commit for details. Many of them are for GNU compatibility, and I’m sure I’ll forget them all. I seem to have issues remembering how to use find(1) successfully.
Sascha Wildner has made it easier to use alternative syntax checking systems as a “lint” make target in DragonFly. His usage of coccinelle, as one of these alternatives, has already found many bugs – just today, for instance.
Is “alternative syntax checking systems” the right phrase for this? I don’t know. ”Correctness checker”? My phrases all sound like something you’d read on a government form.
Reading this HAMMER2 commit carefully shows some future plans: remote cluster control, and the ability to mount nonlocal HAMMER2 volumes. A reminder: those are future plans, not what you can do now.
The i386-only doscmd(1) is gone from DragonFly. I don’t think I ever used it, as other emulators/systems are so prevalent and complete.
Sepherosa Ziehau has been doing a lot of work on packet transmission; far more than I link to here. The end result is startling performance on high-bandwidth links. I’m hoping for benchmarks at some point, but until then, I just wanted to publicly appreciate the work he’s done.
Sepherosa Ziehau has added netblast, a tool originally from FreeBSD that, if I’m reading the commit right, flings packets of a given size at an IP/port of your choosing, for as long as you want.
It’s possible to accidentally truncate your password when using DES encryption and 0×80 in UTF-8 encoding. It’s fixed.
Sepherosa Ziehau has made some changes to SIOCGIFDATA, so if you are using DragonFly-master and pf, you will need a full rebuild. Also pftop, if you use it.
Magliano Andrea Andrea Magliano, a new version of ACPI has been added to DragonFly, acpica-unix 20110527.
If you are running bleeding-edge DragonFly, libpthread was broken for a short period. If you built anything in the last … 12 hours? You may want to rebuild it. If that doesn’t describe you, it’s a nonevent.
It’s funny that I’m reporting a short-term break in bleeding-edge operating system code as any sort of surprise. It shows something about how stable DragonFly-master is most of the time.
Sepherosa Ziehau has been making various updates that conform to standards lately, including “RFC4653 Non-Congestion Robustness (NCR)” and “RFC3517bis“. I’m not familiar with what they do, but you can follow the links and read the RFCs if you are curious.
(Not sure if I got that 3517 one correct…)
Apparently Apache 2.4 has a bug that will cause network stalls when sending data that doesn’t line up with segment size. Sepherosa Ziehau has put in a workaround for the issue. Alternately, you can use www/apache22.