The venerable (from 1979!) program, lpr, has been superseded by CUPS in many installations. Francois Tigeot suggested removing it, but it’s still directly usable in specific situations and easier to just shift out of the way. It’s staying, but it’s interesting to see how it still gets used.
Update: Predrag Punosevac has descriptions of the various tools involved.
I’m planning to branch DragonFly 3.6 this weekend. The actual release will come 2 weeks later. (Ignore what I wrote about a dports installer/image.)
Matthew Dillon wrote a roundup post summarizing all the changes he’s made to DragonFly to improve SMP performance in the last few weeks. He’s removed almost all contention from DragonFly. This means better performance, scaling upward depending on the number of processors.
‘monster’, the system that builds all 20,000 items in dports, can complete the run in 15 hours. Compare this to the 2 weeks it used to take me to build the 12,000 packages in pkgsrc. This is admittedly on different hardware and different packaging systems, but it gives a sense of the scale of the improvement.
The ‘poweroff’ command, the equivalent of ‘halt -p’, has been added based on a suggestion from Robin Hahling.
Matthew Dillon was using poudriere, the dports build tool, on a 48-core system. Poudriere was building all 20,000+ dports, so the machine was quite busy. He decided to get rid of as much contention as possible, and he’s listed all the ways DragonFly’s been streamlined by these efforts. We need to revisit some of our previous benchmarks…
There is a search plugin for Mozilla that searches DragonFly man pages. (Thanks Samuel Greear)
I stole Sepherosa Ziehau’s email subject for the title of this post, because that’s exactly what has happened. Gigabit networking cards under DragonFly will perform very well under extreme load – all of them.
As a followup to news that the git feed of pkgsrc through dragonflybsd.org is not being updated, Max Herrgard wrote out how to fetch pkgsrc via CVS, or tarball, or another git feed. CVS is still the ‘official’ way.
The pkgsrc repository in git for DragonFly is currently frozen. This is because many people have switched over to dports, and also because it’s a lot of work to keep it functional. If you do want to pull newer pkgsrc material, use cvs and grab it from a NetBSD server.
As the message notes, don’t go switching to DragonFly-current right now, cause there’s a lot of new material in there and it may not be quite safe. (There’s an ABI change that will require all new builds of your ports, for instance.)
The Radeon KMS driver from FreeBSD has been imported to DragonFly by Francois Tigeot. It still has problems with ttm, but don’t let that stop you from taking advantage of it.
Google has a post up about the 10th anniversary of Summer of Code, with next year’s version of the event getting some changes – an increase in the students allocated and in the student stipend, and more events. I’m planning to apply for DragonFly, for 2014.
Google is also doing the Code-In, for 13 to 17-year-old students, again. DragonFly participated in the first year (the only BSD to do so), but sat out last year. I’m not currently anticipating DragonFly being involved for 2013, cause of reasons. (It’s a lot of work!)
John Marino has accomplished the major task of updating gdb/kgdb, to version 7.6.1 for DragonFly.
Franco Fichtner recently received commit rights for DragonFly. This is so he could import mdocml, a OpenBSD-originating replacement for groff and man page display. Mdocml has been mentioned before on the Digest, and there’s a downloadable book. (See the more-interesting-than-it-sounds History of UNIX Manpages there too, but I digress.)
One advantage of using mdocml, as I understand it, is that groff is no longer required to view man pages. The only thing left in DragonFly that required a C++ compiler was groff. So, rebuilding could be a bit faster, and a bit less complicated.
Here’s the part that makes me happy: Changes made in DragonFly promptly made it back into NetBSD’s mdocml. Other changes rolled from DragonFly back into OpenBSD, too, and mdocml is in FreeBSD 10, though I don’t have a src change to point at right now. It all circled back around to DragonFly, too. It’s really neat to have a BSD-grown cross-BSD product.
(Incidentally, if you have a Thinkpad and keyboard issues, Franco has a patch for you to try.)
I had this to post, and managed to miss it: Daniel Flores, whose Summer of Code project was Hammer compression, posted a final report.
DragonFly has generally shifted over to dports for 3rd-party software management, away from pkgsrc. Because of that, I haven’t been building binary packages of the quarterly pkgsrc releases. Pierre Abbat asked why on users@, and here’s my explanation of the change.
John Marino has put in a large patch to DragonFly 3.5, updating all sorts of language-related items. As he warns, you will need a full buildworld/buildkernel in a specific order to update. On the plus side, you can now probably use your native language for nvi and for git.
If you want to boot from a Hammer 2 /boot volume, you now can. Hammer 1 never worked well as /boot, though it was technically possible. Hammer 2 will be just fine.
Note that you can’t turn on recently-added disk compression since the bootloader doesn’t understand it, and Hammer 2 is not ready for anything but being worked on. Don’t try it unless you’re ready to be submitting code changes to fix Hammer2.
This will not be a surprise to anyone seeing the work being done, but: All 5 DragonFly/Summer of Code students for 2013 passed, as noted today in emails from Google. It was possibly our best year yet in terms of buckling down and just plain working.
Francois Tigeot posted his work on the KMS driver for Radeon video cards. He’s looking for help since he’s low on time for the immediate future, and this is a project that could benefit everyone. (Well, everyone with the right video card.)
Joris GIOVANNANGELI and Pawel Dziepak both have published final reports for this year’s DragonFly/Summer of Code experience. Both of them say they want to keep working on DragonFly, which is exactly the result I want. There may be more if the other students have time. A final report wasn’t required, but it is good feedback.
Related: Joris is working on Capsicum for DragonFly and published an API document describing how it has worked/will work.
Please welcome our newest committers: Joris Giovannangeli and Mihai Carabas. Joris has already updated bc(1) and dc(1) to match what OpenBSD has. You may recognize Joris’s name from his just-finished Google Summer of Code project for DragonFly, and Mihai Carabas from both this year’s and last year’s Summer of Code.
Matthew Dillon’s committed the work by Daniel Flores on Hammer 2 compression and Mihai Carabas’s vkernel hardware support - both Summer of Code projects. There’s a good amount of detail in the commit messages describing the work and what it changed; I expect more Summer of Code work to be getting committed…
Note: you’ll want to do a full update.
I put together a list of what I’m thinking could be in the next DragonFly release. Going by our regular schedule, that’s a bit more than a month off. Of note: Summer of Code material and defaulting to dports. Follow the thread for more.
DragonFly has two included compilers – GCC 4.4, and GCC 4.7. Traditionally, we switch from one compiler to the other as default, and then replace the old one with a newer release, and so on.
Until recently, dports built almost exclusively using GCC 4.4. John Marino’s switching to GCC 4.7, for a variety of reasons he lists in a recent post to users@. An interesting point that he raises: GCC 4.4 won’t necessarily be replaced with a newer GCC, but perhaps clang?
We’re in the last week of what has been a very good Summer of Code for DragonFly, and here’s the last reports. (We’re missing two, but this is cleanup week, so not much to report)
- Daniel Flores: HAMMER2 compression feature
- Larisa Grigore: System V IPC in userspace
- Pawel Dziepak: Make vkernels checkpointable (updated)
- Joris GIOVANNANGELI: Capsicum (Joris, where’s your report?)
- Mihai Carabas: hardware nested page table support for vkernels
I know this is late; my schedule is a bit messed up. This is the penultimate week!
I’m just going to roll all these updates from Sepherosa Ziehau together into one post, because it’s a lot: He’s updated igb(4) to 2.3.10, updated em(4) to 7.3.8, merged the hardware abstraction layer of those two drivers, enabled TSO on all PCI-E em(4) chipsets, and added support for a whole slew of Realtek chipsets in the re(4) driver. Whew!
If you’ve got a MCP79 NVIDA-chipset board, Sascha Wildner’s commit of Ed Berger’s port from OpenBSD has you covered.
Antonio Huete Jimenez has committed his work on “dirfs”, a filesystem that lets you mount directories from your host machine within the running vkernel environment. It’s a sort of shared folders for vkernels. See the commit message for usage details.
Sepherosa Ziehau has made a number of improvements to TCP in DragonFly – specifically, nonblocking and blocking connect(2) performance. See each of his commits for statistics on how much this has reduced processor use under high load. He has also written up an extensive description of how all this TCP stuff works in DragonFly.
In similar news, he has a nginx patch that delivers a significant performance increase. It may go into nginx itself.
Almost done with this year’s GSoC. It’s been astonishingly… easy? The students are working and the problems are difficult, but there’s been very little in the way of crisis.
avalon.dragonflybsd.org, also known as mirror-master, is the final dragonflybsd.org system to be moved into the new colocated blade server. Your downloads of binary packages or DragonFly images should be speedier.
Remember my recent disk issues? As a side effect of protecting myself, I have a good example of deduplication results.
I have a second disk in my server, with slave Hammer PFSs to match what’s on my main disk. I hadn’t put them in fstab, so they weren’t getting mounted and updated. I got them re-created, but they were nearly full. Here’s an abbreviated df, from which you should be able to tell which drives I have :
Size Used Avail Capacity 929G 729G 200G 78% /slave/slavehome 929G 729G 200G 78% /slave/slavevar 929G 729G 200G 78% /slave/slaveusr 929G 729G 200G 78% /slave/slaveslash
That 78% is how full the Hammer volume was. I turned on Hammer deduplication, since it’s off by default. The very next day:
Size Used Avail Capacity 929G 612G 318G 66% /slave/slavehome 929G 612G 318G 66% /slave/slavevar 929G 612G 318G 66% /slave/slaveusr 929G 612G 318G 66% /slave/slaveslash
It’s a 1 terabyte disk, and I gained more than 10% back – That’s 100g of disk space that I gained overnight. There might be more tomorrow, given that it was all of 5 minutes of dedup work.
This won’t surprise you if you’ve seen previous deduplication links here, like my previous results or some real-world tests. It’s still great. I’d suggest turning it on if you haven’t – hammer viconfig the appropriate PFS and uncomment the dedup line.
Only 3 more Mondays left in the student work part of Summer of Code! Unsurprisingly, it seems the students are mostly in the cleanup phase – as it should be.
I’ll be working on the 3.4.3 release of DragonFly within the next 24 hours, and it should be available this week. I’ll have a list of the bugfixes it contains…
If you’re curious about the hardware being used for the colocated dragonflybsd.org servers (this includes the website, the repository, the mailing lists, dports build machines, etc.), here’s the ‘MicroCloud’ product page. DragonFly’s model was purchased from iXsystems. Apparently those Haswell processors have a fantastic power consumption to performance ratio. (via)
One of the most-requested items for the DragonFly mailing list archives is reverse sorting by date. Mailman, which is what’s being used now for archiving, doesn’t have a ‘native’ way to do that. Has anyone seen a trick/patch to get that to happen? I already patch Mailman to get the message date to show in listings.
I’d be really surprised to find this affects anyone, but it’s possible: some kernel options specific to Cyrix processors have been removed, by Sascha Wildner.
If you look at the reports from students this week, they are mostly “I had bugs and I fixed them and there’s not much to do other than test”, which is the sign of well-planned projects. Here’s the status reports:
The mailing list archives for DragonFly (lists.dragonflybsd.org) have been moved to new hardware. (Yay!) The patch that actually shows date in the listings needs to reapplied, cause Mailman is somewhat stale. (Boo!) I applied the patch and I’m regenerating all the archives now. (Yay!) There’s some garbled messages in the archives that cause a bunch of “no subject” partial messages to be dumped at the end. (Boo!) I’ll manually fix them if I can, someday. (Yay?)
Everyone passed their Summer of Code midterms! Not that this was a surprise; all the students have been consistently working and overcoming problems, but a 100% pass rate makes me happy.
Here’s the status reports:
Joris Giovannangeli, one of the Summer of Code students for DragonFly, posted his thoughts on credential descriptors – have a read. He is working on capsicum and DragonFly, so this is a natural thought process.
If you have a computer with one of the very-very-new Haswell processors from Intel, Matthew Dillon has made some changes that will interest you. They shave off (in the example given) about 20% of CPU power usage without much effect on performance.
killall -T will now kill all processes associated with the current tty, except parents of the killall process itself. It’s a shortcut to “kill all these runaway items I started by accident”.
Thanks to the effort of a number of people, DragonFly (-current) now supports KMS and accelerated video on Intel 915 chipsets. It’s 2D and x86_64 only for now, but it’s much, much better than just using the vesa driver.
Please welcome our newest DragonFly committer: Johannes Hofmann. He earned this by coming up with a significant chunk of DragonFly’s upcoming KMS/915 support, and it’s now easier to just have him work directly than to be constantly committing for him.
It’s week 6, I think, and the midterms are coming up. Here’s the status reports:
Michael W. Lucas wrote a blog post about pkgng and Ansible on FreeBSD. Will it work on DragonFly? We already have pkgng on DragonFly in the form of dports, and Ansible… might work? Please, someone try.
In part of a long thread about dports packages on the users@ list, Matthew Dillon notes that a new set of packages for i386 and x86_64, for 3.4 and for “3.6″ (meaning bleeding-edge DragonFly, even though that’s numbered 3.5) is mostly uploaded. He also notes that a Haswell-processor-based blade server for DragonFly is in the works, so much of the dragonflybsd.org infrastructure is going to move from his house to a datacenter, with the benefits that provides. It’ll also help automate binary package building.
Sepherosa Ziehau added SO_REUSEPORT to DragonFly. I don’t know how the mechanism works, because he didn’t include a description, but he did include a explanation of just how much it reduces CPU usage during as-high-as-physically-possible network load. He even wrote tools to test it more heavily.
I’m late for this, even though the students weren’t. Mea culpa! There’s been a lot of discussion on IRC, in EFNet #dragonflybsd, between the students and various DragonFly developers.
Thanks to the efforts of a large number of people, KMS support is showing up in DragonFly. This supports accelerated video on the new Intel graphics chipsets that seem to show up on many recent laptops.