A reminder based on a question from Pierre Abbat: John Marino isn’t working on 32-bit packages for dports; there’s a volunteer who will, but until the volunteer is ready, 3.7 users will want to build from source.
Here’s how my upgrade from DragonFly 3.4 to 3.6 for this server went.
The system install went normally. I rebooted before performing ‘make upgrade’, as noted in UPGRADING and elsewhere.
I already have dports installed, so a binary upgrade should be possible. I had heard of people with older version of pkg, having trouble getting it to notice upgrades. I rebuilt pkg, and ran ‘pkg upgrade’. A number of the updates coredumped. Here’s one example:
[156/160] Upgrading gtk2 from 2.24.19 to 2.24.19_2...Segmentation fault (core dumped)
After the upgrade, I had two problems: PHP wasn’t working for the website, and some programs would segfault.
The random segfault was fixable by forcing a binary upgrade of all packages. Since there were some programs on the system that were still new enough that the version number was the same as on the remote repository, pkg didn’t upgrade them. Those packages were linked against old versions of system libraries that predated the locale changes in DragonFly 3.6, so they’d crash. Forcing the update for all packages fixed the issue.
The other problem, PHP on the web server, is not new to me. The binary package for PHP does not include the module for Apache. The solution is to build from source with that option selected. I understand that pkg is destined to support (some?) port options in the future. There’s also an immediate workaround for locking it.
However, the port would not build because of a security issue. The binary package installed without any warning. This, I am told, will change to pkg giving you the option to install if you are aware of the security problem, and whether it really affects you. (which is just what I want, yay!)
Anyway, other than the system changes biting me because I didn’t realize some packages weren’t updated, it went very quickly. That is the reason for binary updates through pkg, or at least a major one.
ISA device support is really gone. Well, except for keyboard and some spots where it can’t be be removed. I don’t think I’ve even seen an ISA card in some years…
I had a sometimes-great, sometimes-difficult trip to New York City over the past few days, and while I was there, I met the ball of energy that is George Rosamond of NYCBUG (which is having a huge party right now.) He and I talked for a bit about various aspects of the BSD ecosystem, and one thing he noted was that people aren’t generally aware of all the licenses in use for the different software packages on the system, or even the individual licenses in the system files.
There is an ACCEPTABLE_LICENSES setting in pkgsrc, where software licensed under terms not in that list won’t install. That’s useful, but frustrating, because it keeps people from getting what they asked for – a software install. Something that would be useful – and it could be cross-BSD very easily – would be a license audit summary.
There’s meta-data on every package in FreeBSD’s ports and DragonFly’s dports and pkgsrc and OpenBSD’s port system. Why not say ‘pkg licenses’ in the same way you can say ‘pkg info’, and get a summary of the licenses you have installed in the system? (or pkg_licenses, etc. You get the idea) This wouldn’t prevent people from installing software, but it would give a very quick view of what you were using.
> pkg licenses
Software package License
foo-2.2.26 Apache license
It could be extended to the base system, but I’d like to see this in all the packaging systems as a common idea, in the same way that ‘info’ in a packaging command always shows what’s installed.
Rett Kent has volunteered for maintaining i386 support under dports. Good luck! 3rd-party software management is difficult.
This post from Konrad Neuwirth asking how to do a minimal installation of DragonFly led to this list of all the ‘knobs’ you can set to make your installation smaller, from John Marino. (And your buildworld faster, if that’s appealing to you.) I also pointed at rconfig and PFI, which are criminally underdocumented.
pkg 1.2 is coming out. This brings a number of new features, but as John Marino posted, you may want to delete your old pkg.conf to keep the new version from complaining about an old config file. This upgrade is a step on the way to signed packages, which is a Good Idea.
John Marino posted a possible ‘roadmap’ for DragonFly, now that we’re past the 3.6 release. The thread went on for some ways as it was discussed, including my crazy ideas. Notably, several suggested items have already been tackled – an iwn(4) upgrade has already happened, and an update to bmake, based on John’s vendor branch update instructions.
This is a little old, but Matthew Dillon noted the status of his Hammer2 work a little while ago. Some highlights: he’s intending Hammer2 to be usable on a single host by the time of the next DragonFly release (summer 2014), the Summer of Code project for compression has already been integrated, and he listed different parts of the work that may be interesting for anyone wanting to chip in.
Slightly related: Matt posted some Hammer2 comments on the DragonFly 3.6 release story on Slashdot that may be interesting. Don’t bother reading the other comments; they’ll make your eyeballs bleed.
If you’re planning to run DragonFly in KVM, remember this post from Matthew Dillon, giving the settings he uses. This will save you a bit of time.
If you’re upgrading dports (and you probably are if you are going from DragonFly 3.4 to 3.6), there’s a minor issue in dports, inherited from FreeBSD ports: you need to manually remove perl before upgrading. It’s all of one command, so it’s not a huge burden. Joris Giovanngeli spotted it first.
Eitan Adler is the newest DragonFly committer; you may recognize his name from some previous commits added by others, where he synced up various work between the BSDs.
For those updating from 3.4 to 3.6: there’s an ABI change, so you will have to upgrade all your packages. If you’re using pkgsrc and ready to switch to dports, now’s the time. If you already switched to dports on your 3.4 system, binary packages for 3.6 have already been built and you can use pkg to upgrade.
Also for upgrades from 3.4: You can pull the 3.6 source normally:
git fetch origin
git branch DragonFly_RELEASE_3_6 origin/DragonFly_RELEASE_3_6
git checkout DragonFly_RELEASE_3_6
But there’s a slight change needed for the 3.4 to 3.6 transition: an extra reboot in the build process:
# make buildworld && make buildkernel && make installkernel && make installworld && reboot
# make upgrade
This is all noted in /usr/src/UPDATING and in the release notes, but I’m taking no chances.
As noted on the kernel@ list, it’s tagged but not yet in image form.
John Marino isn’t interested in supporting the i386 architeecture for DragonFly and dports, so he’s not going to actively work on it. (Packages for DragonFly 3.6 are already built, so that’s not a problem for release.) If you feel like taking on a significant but interesting workload, check his message about the work involved.
BSDNow episode 11 is up, with conversations about OpenSSH, FUSE, building an OpenBSD router, etc… and a whole hour of me talking about the upcoming DragonFly 3.6 release and this very Digest, too!
I just finished a whole hour of gabbing on about DragonFly and BSD work in general for BSDNow. Because I am a ninny, I didn’t post something here earlier today so that people would know to watch the livestream. Sorry! However, it should be showing up in the next day or so on the BSDNow site. When it does, I’ll link it.
Not sure why, but there wasn’t a lot of things this week to pick out.
- A short discussion of Perfect Forward Secrecy on pkgsrc-users.
- PC-BSD apparently (used to) play a movie on first boot.
- FreeBSD now has a ‘mini-memstick‘ install option. (a later messages says ~200M in size.)
- FreeBSD has updated aacraid.
- OpenBSD supports the RTS5229 card reader in rtsx(4).
- OpenBSD has updated OpenSSH, and NetBSD has updated. (DragonFly has a fix for the underlying problem.)
- OpenBSD has FUSE support.
Matthew Dillon did some more performance tuning for DragonFly. I’ll just pull a paragraph from the commit message, since that will have more impact than anything I say:
Improves fork/exec concurrency on monster of static binaries from 14200/sec to 55000/sec+. For dynamic binaries improve from around 2500/sec to 9000/sec or so (48 cores fork/exec’ing different dynamic binaries). For the same dynamic binary it’s more around 5000/sec or so.
“monster” is a 48-core machine used for testing.
The venerable (from 1979!) program, lpr, has been superseded by CUPS in many installations. Francois Tigeot suggested removing it, but it’s still directly usable in specific situations and easier to just shift out of the way. It’s staying, but it’s interesting to see how it still gets used.
Update: Predrag Punosevac has descriptions of the various tools involved.
I’m planning to branch DragonFly 3.6 this weekend. The actual release will come 2 weeks later. (Ignore what I wrote about a dports installer/image.)
Matthew Dillon wrote a roundup post summarizing all the changes he’s made to DragonFly to improve SMP performance in the last few weeks. He’s removed almost all contention from DragonFly. This means better performance, scaling upward depending on the number of processors.
‘monster’, the system that builds all 20,000 items in dports, can complete the run in 15 hours. Compare this to the 2 weeks it used to take me to build the 12,000 packages in pkgsrc. This is admittedly on different hardware and different packaging systems, but it gives a sense of the scale of the improvement.
The ‘poweroff’ command, the equivalent of ‘halt -p’, has been added based on a suggestion from Robin Hahling.
Matthew Dillon was using poudriere, the dports build tool, on a 48-core system. Poudriere was building all 20,000+ dports, so the machine was quite busy. He decided to get rid of as much contention as possible, and he’s listed all the ways DragonFly’s been streamlined by these efforts. We need to revisit some of our previous benchmarks…
There is a search plugin for Mozilla that searches DragonFly man pages. (Thanks Samuel Greear)
I stole Sepherosa Ziehau’s email subject for the title of this post, because that’s exactly what has happened. Gigabit networking cards under DragonFly will perform very well under extreme load – all of them.
As a followup to news that the git feed of pkgsrc through dragonflybsd.org is not being updated, Max Herrgard wrote out how to fetch pkgsrc via CVS, or tarball, or another git feed. CVS is still the ‘official’ way.
The pkgsrc repository in git for DragonFly is currently frozen. This is because many people have switched over to dports, and also because it’s a lot of work to keep it functional. If you do want to pull newer pkgsrc material, use cvs and grab it from a NetBSD server.
As the message notes, don’t go switching to DragonFly-current right now, cause there’s a lot of new material in there and it may not be quite safe. (There’s an ABI change that will require all new builds of your ports, for instance.)
The Radeon KMS driver from FreeBSD has been imported to DragonFly by Francois Tigeot. It still has problems with ttm, but don’t let that stop you from taking advantage of it.
Google has a post up about the 10th anniversary of Summer of Code, with next year’s version of the event getting some changes – an increase in the students allocated and in the student stipend, and more events. I’m planning to apply for DragonFly, for 2014.
Google is also doing the Code-In, for 13 to 17-year-old students, again. DragonFly participated in the first year (the only BSD to do so), but sat out last year. I’m not currently anticipating DragonFly being involved for 2013, cause of reasons. (It’s a lot of work!)
John Marino has accomplished the major task of updating gdb/kgdb, to version 7.6.1 for DragonFly.
Franco Fichtner recently received commit rights for DragonFly. This is so he could import mdocml, a OpenBSD-originating replacement for groff and man page display. Mdocml has been mentioned before on the Digest, and there’s a downloadable book. (See the more-interesting-than-it-sounds History of UNIX Manpages there too, but I digress.)
One advantage of using mdocml, as I understand it, is that groff is no longer required to view man pages. The only thing left in DragonFly that required a C++ compiler was groff. So, rebuilding could be a bit faster, and a bit less complicated.
Here’s the part that makes me happy: Changes made in DragonFly promptly made it back into NetBSD’s mdocml. Other changes rolled from DragonFly back into OpenBSD, too, and mdocml is in FreeBSD 10, though I don’t have a src change to point at right now. It all circled back around to DragonFly, too. It’s really neat to have a BSD-grown cross-BSD product.
(Incidentally, if you have a Thinkpad and keyboard issues, Franco has a patch for you to try.)
I had this to post, and managed to miss it: Daniel Flores, whose Summer of Code project was Hammer compression, posted a final report.
DragonFly has generally shifted over to dports for 3rd-party software management, away from pkgsrc. Because of that, I haven’t been building binary packages of the quarterly pkgsrc releases. Pierre Abbat asked why on users@, and here’s my explanation of the change.
John Marino has put in a large patch to DragonFly 3.5, updating all sorts of language-related items. As he warns, you will need a full buildworld/buildkernel in a specific order to update. On the plus side, you can now probably use your native language for nvi and for git.
If you want to boot from a Hammer 2 /boot volume, you now can. Hammer 1 never worked well as /boot, though it was technically possible. Hammer 2 will be just fine.
Note that you can’t turn on recently-added disk compression since the bootloader doesn’t understand it, and Hammer 2 is not ready for anything but being worked on. Don’t try it unless you’re ready to be submitting code changes to fix Hammer2.
This will not be a surprise to anyone seeing the work being done, but: All 5 DragonFly/Summer of Code students for 2013 passed, as noted today in emails from Google. It was possibly our best year yet in terms of buckling down and just plain working.
Francois Tigeot posted his work on the KMS driver for Radeon video cards. He’s looking for help since he’s low on time for the immediate future, and this is a project that could benefit everyone. (Well, everyone with the right video card.)
Joris GIOVANNANGELI and Pawel Dziepak both have published final reports for this year’s DragonFly/Summer of Code experience. Both of them say they want to keep working on DragonFly, which is exactly the result I want. There may be more if the other students have time. A final report wasn’t required, but it is good feedback.
Related: Joris is working on Capsicum for DragonFly and published an API document describing how it has worked/will work.
Please welcome our newest committers: Joris Giovannangeli and Mihai Carabas. Joris has already updated bc(1) and dc(1) to match what OpenBSD has. You may recognize Joris’s name from his just-finished Google Summer of Code project for DragonFly, and Mihai Carabas from both this year’s and last year’s Summer of Code.
Matthew Dillon’s committed the work by Daniel Flores on Hammer 2 compression and Mihai Carabas’s vkernel hardware support - both Summer of Code projects. There’s a good amount of detail in the commit messages describing the work and what it changed; I expect more Summer of Code work to be getting committed…
Note: you’ll want to do a full update.
I put together a list of what I’m thinking could be in the next DragonFly release. Going by our regular schedule, that’s a bit more than a month off. Of note: Summer of Code material and defaulting to dports. Follow the thread for more.
DragonFly has two included compilers – GCC 4.4, and GCC 4.7. Traditionally, we switch from one compiler to the other as default, and then replace the old one with a newer release, and so on.
Until recently, dports built almost exclusively using GCC 4.4. John Marino’s switching to GCC 4.7, for a variety of reasons he lists in a recent post to users@. An interesting point that he raises: GCC 4.4 won’t necessarily be replaced with a newer GCC, but perhaps clang?
We’re in the last week of what has been a very good Summer of Code for DragonFly, and here’s the last reports. (We’re missing two, but this is cleanup week, so not much to report)
- Daniel Flores: HAMMER2 compression feature
- Larisa Grigore: System V IPC in userspace
- Pawel Dziepak: Make vkernels checkpointable (updated)
- Joris GIOVANNANGELI: Capsicum (Joris, where’s your report?)
- Mihai Carabas: hardware nested page table support for vkernels
I know this is late; my schedule is a bit messed up. This is the penultimate week!
I’m just going to roll all these updates from Sepherosa Ziehau together into one post, because it’s a lot: He’s updated igb(4) to 2.3.10, updated em(4) to 7.3.8, merged the hardware abstraction layer of those two drivers, enabled TSO on all PCI-E em(4) chipsets, and added support for a whole slew of Realtek chipsets in the re(4) driver. Whew!
If you’ve got a MCP79 NVIDA-chipset board, Sascha Wildner’s commit of Ed Berger’s port from OpenBSD has you covered.
Antonio Huete Jimenez has committed his work on “dirfs”, a filesystem that lets you mount directories from your host machine within the running vkernel environment. It’s a sort of shared folders for vkernels. See the commit message for usage details.
Sepherosa Ziehau has made a number of improvements to TCP in DragonFly – specifically, nonblocking and blocking connect(2) performance. See each of his commits for statistics on how much this has reduced processor use under high load. He has also written up an extensive description of how all this TCP stuff works in DragonFly.
In similar news, he has a nginx patch that delivers a significant performance increase. It may go into nginx itself.
Almost done with this year’s GSoC. It’s been astonishingly… easy? The students are working and the problems are difficult, but there’s been very little in the way of crisis.