The default version of Python in pkgsrc is going to become 2.7. This will mean the 2012Q1 release will use that version by default. Older versions, meaning Python 2.4 and 2.5, may be going away. At least, that’s how the linked thread started but I’m not totally sure about it as I read farther through.
See the release page for details. This release took longer than normal because of a crazy bug hunt, but the payoff is that this version performs better than ever.
Note: The x86_64 GUI ISO image had a problem due to file size (over 2G); redownload if you’ve had trouble booting it.
I was reading an article about how Tumblr scaled to handle the huge amount of data it’s regularly pushing out. Apparently, it started life as a traditional LAMP stack, but they’ve since moved on – to software packages I have not yet needed to ever use. Being open source software, it all has crazy names. Some of these packages are perfectly familiar to me now, but others are completely new.
Anyway, for fun, I decided to see how many of these sometimes new-to-me packages were present in pkgsrc. I’ll reproduce a paragraph from the story that lists the software they use, and link each one that I found in pkgsrc.
- PHP, Scala, Ruby
- Redis, HBase, MySQL
- Varnish, HA-Proxy, nginx,
- Memcache, Gearman, Kafka, Kestrel, Finagle
- Thrift, HTTP
- Git, Capistrano, Puppet, Jenkins
That’s actually more than I thought I’d find, though I can’t articulate why. Anyway, if any of the names are unfamiliar to you, now is the time to follow up. Redis, for example, looks more interesting to me at a casual glance than the normal NoSQL models I’ve heard about.
If you’ve noticed the main dragonflybsd.org website being down, that’s because both network connections (on different networks!) serving it are down. This makes the website unavailable, and the source code, but you can still pull down images, packages, and the like from avalon.dragonflybsd.org. Hopefully this warning will be out of date soon.
Note: It’s back.
Hey, it’s snowing here! Finally.
- I remember when fractal zooming would bring a desktop computer to its knees. Now, you can do it in a web browser. (via) This exists as a standalone application (x11/XaoS) too.
- I see content from here get splogged, from time to time, and I think that’s what’s happening here. Someone throws “BSD” into a content generator, with ads slapped on top of it? Honestly, I’m not sure what it is. (via)
- Hammer 2 work is starting, as noted earlier this week. Let’s see some details on a similar filesystem project, btrfs. (via)
- You should quit Facebook because privacy etc. you’ve heard it from me before. The arguments are getting more thorough, though.
- Here’s an article from independent game developer Jeff Vogel about serving a niche with your independent work. I like his writing, plus if you squint your eyes and sorta look at that article’s point sideways, you could construe it as relevant for BSD.
- For fun, spot the two things I mention/link to here frequently, in this somewhat hypey article about Tumblr. (via)
- An Economist article about shifting from computer to computer. I read that and realized the one computer constant for me isn’t my desktop – it’s “~”.
- If you ever played games on the Amiga, you may want to watch this movie. It’s clips from a lot of Amiga games. By a lot, I mean an hour and a half of footage total. There were some really advanced games for the time there. (via)
Your unrelated comic link of the week: Shut Up About Cats. The rest of that site’s good too.
Also! On a related link, Venkatesh Srinivas, one of the DragonFly developers, is participating in a bike ride to raise cash for the Ulman Cancer Fund. If you’d like to pledge some money, he’ll feel better as he cycles a ridiculous 4,000 miles across the US.
There’s 7 bug reports to close before releasing DragonFly 3.0. Most of them have dumps to go with them, so each one should be solvable. Please take a look if you have the time and inclination,
There’s a Hammer 2 branch in the DragonFly git repo now, for the next generation of DragonFly’s native file system. Don’t get too excited; as Matthew Dillon explains, it won’t be operational for months, and features won’t get added until much later this year. It’s neat to see the work happening, though, and there’s a new design document to show what’s coming.
If you were thinking about implementing DNSSEC, Michael Lucas did it himself and wrote down his notes. You can read them and either follow along to implement it yourself, or just spectate. The one disadvantage is that it uses BIND 9.9, and I only see 9.8 and 10 in pkgsrc.
I’ve reviewed Michael Lucas’s book here before, so when he offered a chance to read his newest, SSH Mastery, I jumped at the chance. Michael Lucas has published a number of technical books through No Starch Press, and started wondering out loud about self-publishing. This is, I think, his first self-published technical volume.
It’s a very straightforward book. The introduction opens with a promise not to waste space showing how to compile OpenSSH in text. Chapter 2 ends with the sentence, “Now that you understand how SSH encryption works, leave the encryption settings alone.” This stripping-down of the usual tech-book explanations gives it the immediacy of extended documentation on the Internet. Not the multipage how-to articles used as vehicles for advertising, but an in-depth presentation from someone who used OpenSSH to do a number of things, and paid attention while doing it.
It’s a fun read, and there’s a good chance it covers an aspect of SSH that you didn’t know. In my case, it’s the ability to attach a command to a public key used for login. It even covers complex-but-oh-so-useful VPN setups via SSH.
If you’re looking for philosophical reasons to buy it, how about the lack of DRM?
The physical version is not available yet, but the electronic version is available at Amazon (Kindle), Barnes & Noble (Nook), or from Smashwords (every other format ever, including .txt). The Smashwords variety of formats means that you’ll be able to read it on your phone, one way or another; I’d like to see more books that way in the future.
Ulrich Habel wants to update some of the Perl 5 modules in pkgsrc. He published a request for comments, describing what he plans to do for changing some dependencies. He does note that Perl 5 in pkgsrc is at 5.14.2, which is very recent.
I was talking to a relative today who works at a large financial company, which is standardizing on Red Hat Enterprise. I find it strange that Red Hat, which has a lot of money behind it, still ships a years-old and arguably broken version of perl. By using pkgsrc, you’re getting more up-to-date software than people that actually shell out money for the privilege of compiling software.
They are located in the normal place, in .img (USB) and .iso (CD/DVD) formats. I haven’t made the desktop DVD yet; let’s see how these untested versions do…
If you need to use ISDN with DragonFly, speak up now. I think it may get tossed otherwise.
I received an email from No Starch Press about reviewing this book, and my first reaction was to say no. I assumed this was essentially a book about using Bash, and therefore probably not useful to people reading the Digest.
I read it despite my knee-jerk reaction, and I didn’t need to reject it so suddenly. Almost all of the book will apply to any Unix-like system.
My first real experience with something that wasn’t Windows or a Mac was at a summer job during college, sitting in front of a SparcStation 5 editing files and processing data for real estate. Much of my muscle memory about vi and file manipulation dates from then. This book, even though it’s technically for a different operating system, would have been just what I needed. There’s no system administration in the book, just making your way around a filesystem and the tools you need to get results. It’s the kind of skills I think people lose out on when they boot to a graphical interface in Ubuntu, for example, and then never experience these tools.
Negatives: a few areas won’t be of use to most BSD users, like the section on packaging, or the bash-centric instructions in the shell programming area. There’s the occasional off comment, like that OpenSSH originates from “the BSD project”. There’s surprisingly little of this however, and I had to think a bit to write this negative paragraph.
Positives: The book puts the proper focus on some complex but rewarding aspects of command line use, like using vi (alright, vim) and understanding regular expressions. Much of what it covers is the same material I’ve learned to use over time, and explained to others.
There’s clearly two areas to the book; the first half is about using the command line to accomplish work, and the second is about shell programming. Making it at least through the first half will result in being able to work at a prompt with little issue, with the shell programming a nice bonus. It’s not the normal mix of admin tasks and introductory text; it’s about working at the command line. I imagine giving it to new software testers in a lab, or to a Windows user that has to deal with the occasional unfamiliar environment. There isn’t an equivalent BSD-centric book like this, so it wouldn’t hurt a BSD user, either.
It’s available now at the No Starch website.
(David Shao, where are you? If you’re reading this, hop into #dragonflybsd and tell us how things are going with your GEM/KMS work)
‘Live dedup’, where a DragonFly system makes a deduplicative reference to copied data instead of actually copying the data, is now off by default. There’s no definite issue linked to it yet that I know of, but it never hurts to be careful just before a release.
John Marino has pointed out, with a number of examples, that gnat-aux is the best pkgsrc-based compiler for DragonFly right now, in terms of compatibility and support. It’s certainly good news if you are an Ada programmer. He lists some interesting numbers to demonstrate this superiority, though you can’t buildworld with it yet. (gcc 4.4, on DragonFly as part of the system, will do this normally.)
I said posting would be more regular now that the holiday’s over, didn’t I? I lied.
- Here’s a useful idea: a server that allows (Linux) systems with encrypted file systems to boot unattended. I’m not sure how that doesn’t defeat the concept, but actually reading the documentation may help with that. (via, via)
- While on the topic, the EFF says “Encrypt your disk!“. (via)
- The Commodore 64 is 30 years old, for those readers of a certain age who may have had one… I was a Apple ][ kid. (via)
- Aw, thanks.
- “What deduplicating file system should I use?” Well, I can think of an answer.
Happy new year! Regular posting should resume soon now that my holidays are over.
- I like the line, “Please note that BSD manpages are usually better as compare to Linux” [sic] found on this odd page of where to find documentation.
- Hey, this encryption of DNS requests is a good idea. Then again, so is DNSSEC. I’ve done neither.
- Stop using GoDaddy, if you can. There’s plenty of reasons, other than support for SOPA.
- There’s got to be at least one reader who gets this joke.
- If you don’t mind digging through all the comments in this Slashdot article about building a desktop environment, there’s some neat descriptions of different window managers and so on.
- A mild brain teaser to start the year: a regular expression to find prime numbers.
- This is a nice description of just what the Archive Team does. (via)
- The Coming War on General Purpose Computing. Sometimes the stuff on BoingBoing gives me the same irritated feeling as sensationalistic Wired articles, but this one is good to read if you happen to be working on your own operating system. Also, the similar thing with APIs.
- This “best tech writing of 2011” summary on Verge (via) led me to this excellent article: “The Web Is a Customer Service Medium“. There’s lots more reading in that summary.
- I’ve seen this mentioned before, but now it’s with a graph so it’s better! On the continuing decline of the GPL.
- OK, I admit graphs are not always a good idea. (via)
- Trivium, from which I yoinked that last link, also has an blog from its author, Chris Neukirchen. It’s not updated often but there’s some entertaining sysadmin tidbits on there, such as going all-ed, or zsh tips, or Why I use the MIT license.
Your completely unrelated link of the day: Tiny Legs of Fire. (video) Worth it for the origin of Beardslap.
(Sorry about the giant text block. This isn’t as readable as I’d like.)
Chris Turner got it working on i386, at least, and his post will help you do the same. I don’t know if these changes have made it through to pkgsrc or for x86_64 yet.
I’m linking to this small discussion about licensing and its documentation in pkgsrc, just because these paragraphs, out of context, are good for any pkgsrc user to know.
The freeze for pkgsrc-2011Q4 has started. No updates to pkgsrc, other than for security, for the next two weeks.
The last quarterly release of pkgsrc for the year is scheduled for the end of this month. This means the freeze, where only bugfixes are applies, will be starting on the 17th.
I do this almost every year, with a little bit more every time. Check those previous years for non-comics/books, cause that’s most of what I’ve seen lately. I’ve recently seen a number of comics lists:
Last week was low on links, but this week is great! I hope you have some time set aside.
- This article “The Strange Birth and Long Life of UNIX” has a picture of a PDP-11. I don’t know if I ever actually saw one and knew it before. (via)
- Also from the same place: Window Managers Bloodlines.
- Anecdotal, but probably true. (via luxh on EFNet #dragonfly)
- nginx is the new cool and unpronounceable web server these days, apparently. Michael Lucas covers how to transition static Apache sites over to it.
- This PDF showing slides from the recent NYCBUG presentation by Ike Levy, titled “Inappropriate Cloud Use”, is entertaining, and makes a good point. Cloud computing is cheap on a per month basis, but since it’s a reoccurring cost, it can cost a surprisingly large amount in the long run. (via)
- Hey, a patch for DragonFly (and other BSD) support in Google’s leveldb.
- “Don’t Be a Free User” (via) The last paragraph is the best.
- An expanded grep and diff. ‘grep’ and ‘diff’ have been present for so long, and people understand what they do, generally, that new tools get named after them just because the concept is ingrained in people’s minds. Note that I said “generally”, as regular expressions can be difficult. (via)
- A lot of people don’t realize how they infringe on copyright. This writeup describes something I’ve seen for years: people think a disclaimer that effectively says “I’m infringing but I’m doing it with the best of intentions” makes a difference. It doesn’t.
- So this is what that Xerox Star GUI interface looked like. You know, the ‘first’ desktop GUI. (via) Also, there was some advanced stuff in 1968.
- I like this indicator light setup. (also via luxh on EFNet #dragonflybsd) There’s some other interesting old computer stuff at that site too. I wish there still were computers like these.
- While we’re talking about old things with a certain feel to them, why not Battersea Power Station? Here’s some pictures. (via)
Your unrelated link of the day: Since we’re talking about old things and environments, why not look at some pictures of my workplace?
Two tips for working with pkgsrc, derived in part from this mailing list post on users@ (follow the thread) and from my own experience. If you put
- You can clean up any leftover package building files by deleting the files in that directory and leave your pkgsrc files untouched.
- You can have a read-only /usr/pkgsrc, which means it can be shared over NFS (or SMB?) between multiple machines, DragonFly or otherwise.
Happy (post) Turkey Day for the U.S. readers! A light link week this week.
- Facebook is bad for the Internet. ‘Gaslighting’ is a new term to me. As that article points out, I can’t even put my posts to the Digest onto Facebook in any sort of automated way. Facebook suggests that of course I’d love to retype them all by hand. That’s not realistic. Facebook doesn’t want any sort of useful external link to be visible to their customers. Customers isn’t actually the right word; the customers are the advertisers. What would be a better word for the users? Crop?
- “the internet is above and beyond all else a resentment machine.“ It’s a very long essay that points out people are confusing brand identity with personal identity. (via)
- You know what would be good? More conversations about games on BSD, cause it could use some attention. Oh hey there you go.
- A Dragonfly lamp (via Julian Gehtdichgarnichtsan)
Your unrelated link of the week: Animals Talking In All Caps. It is what it says it is.
Francois Tigeot has updated his PDF of Postgres benchmarks with some OpenIndiana results. They’re crazy high, though he reported some freezes too, as with Linux.
When building world and kernel on DragonFly, /usr/obj is where the work files get placed. This can eat a bit of space, but it can be safely deleted. If you keep the files around, subsequent rebuilds can be done faster with a quickwork/quickkernel, but this may not matter to you.
(This was answered on the mailing lists by Max Herrgaard, but I don’t have a link to his reply – sorry!)
DragonFly now uses Redmine for bugs.dragonflybsd.org. This means that the bugs@ and submit@ lists have can still be read by anyone, but to post a new bug or patch, or reply, you need to be registered on the bug tracker itself. You don’t have to be subscribed to the mailing list to use the web interface. See the bugs@ and submit@ announcements for other details.
Juan Francisco Cantero Hurtado has been working with clang and DragonFly, along with Sascha Wildner. DragonFly mostly compiles using clang, with lib/citrus being (the only? one of?) the last holdouts. Juan Francisco Cantero Hurtado detailed how to test it out using clang 3.0 in case someone else wants to help solve this.
The two things that make my day! The work on DragonFly-current has led to some significant speed improvements. So good, that Samuel Greear’s post on OSNews.org links to graphed results from him and from Francois Tigeot (multi-page PDF) showing the results from pgbench.
The results show a jump in multi-core/processor numbers that vastly exceeds DragonFly 2.10′s performance, and is comparable to FreeBSD 9/10. Here’s some of what did it.
The host leaf.dragonflybsd.org has been upgraded to new hardware. This is the machine used for anyone who wants to develop on DragonFly, so there’s a good performance boost there for developers. It also hosts bugs.dragonflybsd.org, which should be working again soon.
In DragonFly, there’s only a few places C++ is used. If you wanted to make sure DragonFly was pure C, Samuel Greear lists those remaining nooks and crannies.
Almost all the packages in pkgsrc support non-root installation now… except these last 31. I recall something about their removal by the next quarterly release if they still don’t work, or maybe just after. Jump in if one of these packages is useful to you.
Some cleanup in the CVS -> git process wasn’t happening, so if you have been using pkgsrc 2011Q3 from git (i.e. via make in /usr), re-pull to make sure you have everything.
(The post noting this seems to have been eaten by the mailarchive… that’ll be replaced.)
There’ll be some brief outages this week as a few of the dragonflybsd.org machines are upgraded. The new machines will be 64-bit DragonFly, and have 16G of RAM. RAM is crazy cheap these days. I’m continually dumbfounded by it.
Well, they’re still available, but you don’t want them in your config any more because they can slow you down. This will only affect you if you are running binary files from DragonFly 1.2 or earlier, or… I guess a 4.3 BSD binary? From 1986? I’m sure there’s some other reason for it to be there.
Samuel Greear, Jan Lentfer, and others are looking at Postgres scaling on DragonFly. The work they are doing isn’t in the tree yet, but here’s a graph showing some of the performance differences.
Francois Tigeot does something very useful: he monitors the resource usage on his systems, and tracks how it changes over time. Because of that, he noticed that the recent VM changes in DragonFly have made quite a difference in memory usage. (See the green area in the attached chart, around week 42.)
Michael Lucas is building jails on DragonFly, and his story of doing so works pretty well as a how-to guide.
DragonFly’s now on the BuildFarm list of Postgres test systems. (via Jan Lentfer in IRC)
Samuel Greear graphed the performance differences for Postgres and MySQL on DragonFly, before and after the recent VM changes. Note that 1: this was done a little while ago, so I think the performance difference would be even greater now, and 2: this was graphed versus the already-performing-better 2.12, not the current stable release of 2.10.
There’s a rare crash in DragonFly 2.10, where applications would segfault. The system would run find. This is apparently more likely to happen in 2.12, though reports on this vary. It’s real, though.
Matthew Dillon went looking for this bug, and happened to roll back vm_token, the last lock in DragonFly that presented a serious impediment to multiprocessing. It’s a big patch. It fixes the problem, which is great! It also happens to make DragonFly buildworlds almost twice as fast depending on the number of cores in the system.
Holy crap we want to get that out… but it makes some significant changes to the system and needs to be tested. So, the next release probably won’t be for a few weeks.
If you want to help, build master and do something with it – move data, run server programs, whatever. Report crashes. This performance improvement is worth working for.
They aren’t really release candidates per se, just “images I built from the 2.12 branch”, but they are available for testing.
There’s only two commits, already in DragonFly-current, to add to 2.12 before it’s clear of all listed release requirements. And maybe binary package builds… which I’m about 2/3 of the way through.
Dennis Ritchie, one of the people behind UNIX and the C language, has died. (via skullY on #dragonflybsd on EFNet) Look at his Bell Labs web page for some details on his history. The death of Steve Jobs will get a lot of media attention, but I’d argue that Ritchie affected more computers in far more ways.
There’s only one multiprocessing bottleneck left in DragonFly: vm_token. Matthew Dillon’s working on removing it, and he’s been testing his initial results on a 4-core machine and a 48-core machine, using heavily parallelized buildworlds to test concurrency. He’s posted the results, showing an initial speedup of up to 30%. This definitely isn’t going to make it into 2.12, but it’s looking good already. Keep in mind these are improvements on top of the performance graphed here yesterday.
Venkatesh Srinivas sent along a graph of his nmalloc testing that shows mysql threading performance on DragonFly, from slightly over a year ago. Both graphs were done on a 4-core system, though I don’t know if the specs are comparable, so the curve is important. Look at the just-posted curve for comparison. That’s how much things have improved.
In fact, here’s a cheesy overlay, cropping the more recent results and laying the old ones on top of it. The black lines are the year-ago performance, and the colored lines are the performance now.
Samuel Greear has graphed out the performance of both MySQL and Postgres on DragonFly 2.12 as you add threads. There’s a very nice correlation on performance and number of cores. For comparison, there’s this old test from 2007 which shows uniprocessor performance to be good but not improved by adding cores. The tests were on completely different hardware, so the actual curve of the graph is the telling point.
As he points out in his post, excellent multiprocessor performance is arriving on DragonFly, without any catastrophic shifts or destabilizing changes.
The 2.12 branching generated a list of every DragonFly commit since 2.10, grouped by committer. Good to browse through. Try to ignore the part where it shows the measly 4 things I did, with poorly constructed commit messages.
It’s not the 2.12 release yet – just the initial branch of 2.12. This will become the release version of 2.12 in a few weeks.
The Open Source Business Resource, linked here before, has become the Technology Innovation Management Review, or TIM Review. Conveniently, the editor is named Chris, not Tim, so nobody will get confused. It’ll still cover open-source software, but it’ll also
“share the spotlight with topics such as managing innovation, technology entrepreneurship, and economic development”
The first relaunched issue will be out in October.
A position opened up for a junior systems administrator at my workplace. You have to be willing to live near Rochester, NY, administrate a mix of Windows and unixy machines, do desktop support, and network management. (e.g. everything possible) The work environment is neat, informal, and somewhat adverse. I’ll have a job description soon, I hope.