Poudriere is the tool for building all of ports/dports, and Michael W. Lucas has written up his experience using it to build a custom ports set. He’s doing on FreeBSD, but if you ignore the geom-specific parts, it should generally apply to DragonFly.
If you are upgrading packages on your DragonFly 3.6 system, and you have docbook installed, there’s an extra step needed because of the moving around of several docbook packages. If you don’t have docbook installed – nothing to see here.
We’ve got Go builders running for DragonFly, but nobody actively maintaining Go itself on DragonFly. The dports version builds, but there’s a Go release coming up and having native support would be much better than relying on chance FreeBSD build compatibility.
The current error as I type this is a TLS problem that sounds like a simple fix, if only I knew where it was.
Brad Fitzpatrick showed up on the users@ list and mentioned that for DragonFly to be supported in Go, it needed to show up in the Go Dashboard with building reports. I now have the Go builder running on pkgbox32/pkgbox64.dragonflybsd.org. Check the builder page to see status.
Note: Installing the port of Go from Dports works just fine; this is the mechanism for testing Go on a per-commit basis for the people who work on Go – so a ‘fail’ notice on the builder page doesn’t necessarily mean anything, unless you are developing Go itself. This may already be clear to you.
Here’s how my upgrade from DragonFly 3.4 to 3.6 for this server went.
The system install went normally. I rebooted before performing ‘make upgrade’, as noted in UPGRADING and elsewhere.
I already have dports installed, so a binary upgrade should be possible. I had heard of people with older version of pkg, having trouble getting it to notice upgrades. I rebuilt pkg, and ran ‘pkg upgrade’. A number of the updates coredumped. Here’s one example:
[156/160] Upgrading gtk2 from 2.24.19 to 2.24.19_2...Segmentation fault (core dumped)
After the upgrade, I had two problems: PHP wasn’t working for the website, and some programs would segfault.
The random segfault was fixable by forcing a binary upgrade of all packages. Since there were some programs on the system that were still new enough that the version number was the same as on the remote repository, pkg didn’t upgrade them. Those packages were linked against old versions of system libraries that predated the locale changes in DragonFly 3.6, so they’d crash. Forcing the update for all packages fixed the issue.
The other problem, PHP on the web server, is not new to me. The binary package for PHP does not include the module for Apache. The solution is to build from source with that option selected. I understand that pkg is destined to support (some?) port options in the future. There’s also an immediate workaround for locking it.
However, the port would not build because of a security issue. The binary package installed without any warning. This, I am told, will change to pkg giving you the option to install if you are aware of the security problem, and whether it really affects you. (which is just what I want, yay!)
Anyway, other than the system changes biting me because I didn’t realize some packages weren’t updated, it went very quickly. That is the reason for binary updates through pkg, or at least a major one.
Odds and ends for the quieter holidays.
- Hubert Feyrer spotted this video interview of Amitai ‘schmonz’ Schlair about NetBSD.
- OpenBSD has tmpfs.
- PC-BSD has made it through a pkg upgrade.
- pkgsrc is frozen until at least the end of the month, for pkgsrc-2013Q4.
- OpenBSD wants to shift electrical costs. (via)
- The DiscoverBSD weekly roundup.
- Managing custom ports. (can apply to dports too)
- Building tcsh on 4.3BSD-Quasijarus. This led me to…
- 4.5BSD. An ambitious project.
- A pfSense video review.
- Steryana Shopova is this past week’s Faces of FreeBSD.
- OpenBSD had a head start on not trusting RNGs.
- OpenBSD has a new vioscsi(4) driver.
- Michael W. Lucas’s books are available through OpenBSD.
- FreeBSD Kitten. (via NYCBUG)
I had a sometimes-great, sometimes-difficult trip to New York City over the past few days, and while I was there, I met the ball of energy that is George Rosamond of NYCBUG (which is having a huge party right now.) He and I talked for a bit about various aspects of the BSD ecosystem, and one thing he noted was that people aren’t generally aware of all the licenses in use for the different software packages on the system, or even the individual licenses in the system files.
There is an ACCEPTABLE_LICENSES setting in pkgsrc, where software licensed under terms not in that list won’t install. That’s useful, but frustrating, because it keeps people from getting what they asked for – a software install. Something that would be useful – and it could be cross-BSD very easily – would be a license audit summary.
There’s meta-data on every package in FreeBSD’s ports and DragonFly’s dports and pkgsrc and OpenBSD’s port system. Why not say ‘pkg licenses’ in the same way you can say ‘pkg info’, and get a summary of the licenses you have installed in the system? (or pkg_licenses, etc. You get the idea) This wouldn’t prevent people from installing software, but it would give a very quick view of what you were using.
> pkg licenses
Software package License
foo-2.2.26 Apache license
It could be extended to the base system, but I’d like to see this in all the packaging systems as a common idea, in the same way that ‘info’ in a packaging command always shows what’s installed.
Rett Kent has volunteered for maintaining i386 support under dports. Good luck! 3rd-party software management is difficult.
pkg 1.2 is coming out. This brings a number of new features, but as John Marino posted, you may want to delete your old pkg.conf to keep the new version from complaining about an old config file. This upgrade is a step on the way to signed packages, which is a Good Idea.
If you’re upgrading dports (and you probably are if you are going from DragonFly 3.4 to 3.6), there’s a minor issue in dports, inherited from FreeBSD ports: you need to manually remove perl before upgrading. It’s all of one command, so it’s not a huge burden. Joris Giovanngeli spotted it first.
John Marino isn’t interested in supporting the i386 architeecture for DragonFly and dports, so he’s not going to actively work on it. (Packages for DragonFly 3.6 are already built, so that’s not a problem for release.) If you feel like taking on a significant but interesting workload, check his message about the work involved.
Matthew Dillon wrote a roundup post summarizing all the changes he’s made to DragonFly to improve SMP performance in the last few weeks. He’s removed almost all contention from DragonFly. This means better performance, scaling upward depending on the number of processors.
‘monster’, the system that builds all 20,000 items in dports, can complete the run in 15 hours. Compare this to the 2 weeks it used to take me to build the 12,000 packages in pkgsrc. This is admittedly on different hardware and different packaging systems, but it gives a sense of the scale of the improvement.
As a followup to news that the git feed of pkgsrc through dragonflybsd.org is not being updated, Max Herrgard wrote out how to fetch pkgsrc via CVS, or tarball, or another git feed. CVS is still the ‘official’ way.
DragonFly has generally shifted over to dports for 3rd-party software management, away from pkgsrc. Because of that, I haven’t been building binary packages of the quarterly pkgsrc releases. Pierre Abbat asked why on users@, and here’s my explanation of the change.
Since there’s a newer set of dports binary packages uploaded, I thought I’d spend my weekend upgrading, to catch up.
And that was it. Well, not really. I had to dump and restore my Postgres databases, cause of the switch from 9.0 to 9.2 as default. I had to build php5 from source to get the Apache module. Those two things together took longer than the entire download and upgrade of the rest of my system – some ~200 packages?
Michael W. Lucas wrote a blog post about pkgng and Ansible on FreeBSD. Will it work on DragonFly? We already have pkgng on DragonFly in the form of dports, and Ansible… might work? Please, someone try.
In part of a long thread about dports packages on the users@ list, Matthew Dillon notes that a new set of packages for i386 and x86_64, for 3.4 and for “3.6″ (meaning bleeding-edge DragonFly, even though that’s numbered 3.5) is mostly uploaded. He also notes that a Haswell-processor-based blade server for DragonFly is in the works, so much of the dragonflybsd.org infrastructure is going to move from his house to a datacenter, with the benefits that provides. It’ll also help automate binary package building.
It looks like OpenJDK7 works in pkgsrc for DragonFly, thanks to Ryo ONODERA, and I think it’s working in dports too.
FreeBSDNews.net has a nice summary up of video from all (?) the presentations at BSDCan 2013. Of particular interest to DragonFly users: a video about pkg, the tool used for package maintenance in dports. In this presentation, it’s talking about use on FreeBSD, but the future stuff applies to DragonFly too.
Since dports uses FreeBSD ports as a base, adding something to FreeBSD ports means it will show in dports, too. However, it doesn’t have to go that way. It’s possible to have dports packages that exist only in dports. If you have changes to a port that make it compile on DragonFly, that can be added too. For all of that, go to the dports issues page on GitHub.
I pointed out in my converting-to-dports post from yesterday that I had to download dports and build pkg by hand in order to install binary packages. This was because my DragonFly system was upgraded from 3.2 to 3.4 and therefore didn’t have pkg installed.
John Marino has added a ‘pkg-bootstrap’ option to /usr/Makefile, for fixing exactly that problem. It downloads a static version of pkg, which then lets you upgrade to the full pkg and install binaries as you’d expect.
I changed shiningsilence.com over from pkgsrc to dports over the last 48 hours or so. Here’s how it went, in a series of bullet points:
- I had to download dports source and build the pkg tool by hand; since this system was upgraded from DragonFly 3.2 to DragonFly 3.4, pkg wasn’t automatically present as it would be for a new installation.
- I took the output of ‘pkg_info’ and culled it down to the applications I knew I used, and that formed my ‘to-install’ list for dports. That worked in a very straightforward way.
- It took so long mostly because of two things: I was also dealing with an email problem at my workplace, which usually took precedence. Also, I had several applications that I had previously installed by hand and needed to reconfigure to work as a dports item.
- Installing from binaries is really fast! Really, the dports part of this was possibly the most brief.
- The only thing I needed to compile from source was php, in order to get the Apache plugin. I’m sort of surprised the option isn’t on by default.
- Using ‘pkg search packagename’ is a good idea, because ‘pkg install’ can pick up multiple versions of a package. e.g. ‘pkg install mysql-server’ selects mysql-server51, mysql-server55, and mysql-server56. You probably don’t want to install all three. Or even one, depending on your opinions.
- Overall, it went more easily than I had expected, given it only had half of my attention.
I’m switching this server from pkgsrc to dports. No post while I fight with old, stale configs and etc.
Are you using it and unable to upgrade to KDE4 for a specific reason other than aesthetic preference? You should check this thread about support for 3.5, at least in dports.
There’s more download statistics on dports and pkgsrc packages, from Francois Tigeot. There’s a heck of a lot of dports activity, though there’s probably much more pkgsrc building from source than this would report on. So, not necessarily representative of actual numbers, but an interesting ratio none the less.
New builds of dports have been uploaded and updated, for x86_64 and i386. (x86_64 was already done; I linked the note about i386) This means you can change PACKAGESITE in /usr/local/etc/pkg.conf to point at LATEST instead of RELEASE and get newer packages. ’pkg upgrade’ is all it takes, with dports.
‘william opensource4you’ posted a summary of the steps he took for setting up a DragonFly system with XFCE4, using dports. It’s pretty straightforward, and thanks to dport’s binary nature, should be exactly reproducible.
John Marino has a concise explanation of why dports mostly uses gcc 4.4 still to compile, even if you’re building DragonFly itself with the default 4.7. It’s a reason to not use NO_GCC44 – yet.
Eric Radman sent along a plug for a utility he is working on called entr(1). The desciption is “Run arbitrary commands when files change.” The site for it has several nifty examples – run make when *.c files change, or convert Markdown files to HTML as soon as they are modified. The really nice thing about it is that it’s perfectly BSD-friendly, and uses kqueue, but will also work on Linux. This beats the “This runs on the one flavor of Linux I use, in one particular shell!” approach I’ve seen from some other developers. See the reddit discussion of it for comparisons to inotify.
No, it’s not in pkgsrc/ports yet.
DPorts is based off of FreeBSD’s ports, but it’s possible to add software packages to it that don’t exist in FreeBSD’s ports system and have them build as any other packages. This is briefly detailed in this GitHub bug report, along with a number of the ports that already exist that way.
John Marino has posted about the state of dports: over 19500 ports built, build logs available, and patches to add even more can be sent through github. XFCE4, KDE3, and KDE4 are building, though he could use some help with GNOME2.
Man, I’m stretching it to make that “Over nine thousand!” joke, now.
I hope you like reading; there’s some very meaty links this week. Go get a cup of tea and settle in. You drink tea, don’t you? You ought to.
- Reading about KDE’s repository near-meltdown makes me think we need more checks for DragonFly. We have the advantage of Hammer, of course, which would help in the same way that the linked article names ZFS as a ‘fix’. (via multiple places)
- We know that Apple will reject apps it disagrees with. Google also will do so. Has there ever been a program rejected from pkgsrc or (FreeBSD/OpenBSD) ports on content grounds? Not that I know of – anyone remember differently? I’d argue that’s a favorable point for the BSD packaging systems, though it may just be that no application has tested those boundaries yet.
- Portscanning all IPv4 addresses on the planet. Possibly the largest distributed effort ever? The detail in the maps and returned services is especially interesting. (via)
- Scale Fail, a Youtube video of a 2011 talk about screwing up your services. Mostly about the humor, but the underlying points are valid. (via #dragonflybsd IRC)
- There’s still improvement possible to fsck, apparently based on this. That’s UFS2 fsck.
- What is your most productive shortcut with Vim? A very thorough explanation of verbs, marks, and registers. Holy cow, I wish I had known about ‘: … v’ before. It’s long, but worth it. (via)
- Matthew Garret’s description of Secure Boot vs. Restricted Boot with UEFI, (via a coworker who went to Libreplanet 2013). I’m still not sure what DragonFly will need to do about this.
- I missed mentioning this earlier: 20 years of NetBSD. We’re coming up on 10 soon.
- Dragonfly drones. Unrelated except for name.
- That guy who starts to froth madly every time BSD is mentioned on Phoronix is still there (see comments).
- Mainframe computer supercut. (via)
Your unrelated comics link of the week: Tom Spurgeon of the Comics Reporter asked people for their lists of webcomics that could go in a ‘Hall of Fame’. The resulting list is a lot of really, really good material. Go use up a few hours reading.
I saw this Hacker News post and figured I should emphasize: pkgsrc is still going to be available in the 3.4 release of DragonFly; we’re not suddenly switching to dports. I don’t want anyone to think they’re going to have to rip out all their packages and go to a new, untried system, all at once.
I am all over the place with links this week – some of them pretty far off the path. There’s a lot, too, so enjoy!
- Puctuation obscurantism, punctuation humor; I like it all. (via)
- Exporting your git repository. Found while looking for something else.
- I want CTRL-D at a terminal to make something like this to happen.
- Visual Representation of Regular Expression Character Classes. I like visual ways of classifying complex data.
- Speaking of which: Anatomy of Data. Not sure how I found it.
- Digital Files and 3D Printing – In the Renaissance? The title sounds a bit linkbaity, but the story of the 14th century map designed to be recreated with a graphing tool is pretty neat.
- Postgres: The Bits You Haven’t Found. Advanced/odd Postgres usage. (via)
- Breaking your arrow keys is the latest idea in improving Vim usage.
- PC-BSD is moving to a ‘rolling release’ format, and also using the new pkg tools that are also in DPorts. Historic details on this new setup are available.
- Fred, taking off.
- Ten hours with the most inscrutable game of all time. I like the idea of Dwarf Fortress more than I actually like playing it. I’m somewhat afraid of it. It looks like this sounds.
- That last comparison wasn’t necessarily fair, but it was fun.
- If I’m going to talk about music like that, I should link Ishkur’s Guide to Electronic Music.
- The Wizard of Pinball. I just want my own standup pinball or arcade cabinet game. Yes, yes, I know, MAME cabinet.
- Appropriately this week, “Ball Saved”, page 1 and page 2 of a 2-page comic about pinball.
- UnReal World, an Iron-Age roguelike. Apparently pretty brutal, and two decades in development. Runs on several platforms, but not BSD. (via)
- You Are Boring. Some of the ‘boring’ items made me laugh. (via)
- The first review of Michael W. Lucas’s Absolute OpenBSD, Second Edition is available.
Your unrelated link of the week: I’ve already been offbeat enough in this Lazy Reading; I don’t have anything else.
Here’s 3 recent and different commits to DragonFly that I’m commenting on all at once:
- Peter Avalos upgraded libarchive in DragonFly to 3.1.2, with a note of the changes. An ordinary and appreciated update.
- Sascha Wildner updated the ISO639 file to include the newest update: “Standard Moroccan Tamazight”. There’s no particular utility to that; I just like saying “Standard Moroccan Tamazight” out loud.
- Work on poudriere, the utility for bulk-building DPorts packages, has caused some nice speedups for DragonFly in extremely stressful situations. See one of Matthew Dillon’s recent commits.
I really wish the other BSD projects would include commit lines in the mail message subjects, so it was easier to catch things like these.
John Marino’s DPorts project, mentioned here briefly before, is interesting. I had two separate people ask me how it works, so a better explanation is in order. I’ve tried it out on a test machine over the past few weeks.
Dports is an effort to use FreeBSD’s ports system as a base for DragonFly, and the pkg tool as a way to manage binary packages built from DPorts. This is complicated, so I’ll explain each part in order.
- FreeBSD ports are a FreeBSD-specific collection of software installation files that automate building 3rd-party software on FreeBSD. You’ve probably already heard of them. (Note there’s no mention of DragonFly.)
- DPorts is a collection of files that map to existing FreeBSD ports, and contain any changes necessary to make that port also build on DragonFly. Many of those programs build without changes on DragonFly. DPorts builds from source.
- pkg is used for package management, and is usable on FreeBSD and on DragonFly. The binary packages produced from building with DPorts can be installed from remote locations and managed separately using pkg, so that software upgrades and installation can be performed with binaries only. (It’s much faster that way.)
Every port seen in DPorts is known to build on DragonFly. John Marino adds a port only after it builds successfully, using poudriere as a bulk software tool. Ports are only updated to a newer version when that newer version builds, too, so once something arrives in DPorts, it should never break from being updated at some point in the future.
To use DPorts, you need two things:
- DragonFly 3.3 or later, though 3.3 is the most recent right now.
- You need to rename /usr/pkg so that your existing pkgsrc binary programs don’t get accidentally used while working with DPorts, causing confusion. If anything goes wrong with DPorts when you are installing it and you want to go back, remove all the DPorts packages and rename /usr/pkg back to normal.
(Don’t confuse pkg, the management tool, with /usr/pkg, the normal installation directory for pkgsrc. ) For the installation of the base port files:
cd /usr make dports-create-shallow
If you’ve already renamed your /usr/pkg directory, git won’t be in your path any more. You can instead download a tarball and unpack it, which also happens to be possible automatically via that same Makefile.
cd /usr make dports-download
Downloading via git is fastest, so if you do need to use the tarball via make dports-download, build devel/git, delete /usr/dports, and then pull it again with make dports-create-shallow. This all comes from John Marino’s Github site for DPorts.
DPorts doesn’t use pkg_info, pkg_add, and the other tools traditionally seen on DragonFly for pkgsrc. Instead, package management is done with pkg. Use pkg info, pkg install, pkg remove, and pkg update to list, install, delete, and upgrade various packages on your system. Packages built from source or downloaded as prebuilt binaries are managed the same way, using these tools.
Since DPorts doesn’t update a package until it gets a successful build, and installations are of successfully built binary packages, upgrades with prebuilt packages should always succeed. Since they’re binary, they should be fast. There’s a lot of ‘shoulds’ in this sentence, but these are reasonable suppositions.
What about pkgsrc?
Pkgsrc and DPorts shouldn’t be used at the same time, since one system’s packages may be at different versions but still get picked up during building for the other system. That’s about it for restrictions.
I intend to try building an experimental release of DragonFly with DPorts, to see if all the right packages can be added, but no guarantees. DPorts is brand new and does not yet have a repository for downloading packages, so the normal caveats apply; don’t install it on a mission-critical machine, and be ready to deal with any surprises from using it if you do try it out.
What packages are available?
Browsing the Github repo will show you all listed packages. More complex packages like xorg, openjdk7, and libreoffice install, as does xfce. Parts of KDE 3 and KDE 4 are in there. (I haven’t tried either.) I’m not sure about Gnome, but I don’t think anyone ever is. There’s no vim, but there is emacs.
That’s just what I see at this exact minute. It changes daily as more packages are built. Changes from DragonFly builds are sometimes relevant to the original FreeBSD port, so there’s benefits for everyone here.
Try it now if it has all the packages you need, or wait for a binary repository to be created to speed things up. Remember, this is a new project, so a willingness to deal with problems and contribute to fixes is necessary.