Packaging discussion log

Included in this entry is a log from #dragonflybsd where several folks talk about the packaging system proposal – I’ve cleaned it up a bit and I present it for your perusal.

Feb 25 15:37:34 corecode> comment on content
Feb 25 15:37:35 corecode> :)
Feb 25 15:37:47 coolvibe> we did ;)
Feb 25 15:38:13 corecode> no you didn’t
Feb 25 15:38:22 corecode> can’t be that there are no mistakes in there
Feb 25 15:38:46 coolvibe>@coolvibe> corecode: good stuff anyways
Feb 25 15:39:03 corecode> bah
Feb 25 15:39:12 coolvibe> I also said something about the openssl stuff
Feb 25 15:39:22 debugger> corecode: A nice this to have is to the pkg system track which package depends (statically) on another one, so we can recompile those too (when the pkg they depend is updated)
Feb 25 15:39:28 coolvibe> corecode: hmm, as for supporting e.g. varsyms or not, maybe it should reference a capability list that tells ithow to use some special facilities?
Feb 25 15:39:33 debugger> errr, s,this,thing
Feb 25 15:39:33 coolvibe> corecode: the crypto signatures thing is pretty easy to solve. Just have the OTS complain a lot when it can’t find supporting programs a lot, and offer to install OpenSSL :)
Feb 25 15:39:53 corecode> coolvibe: OS caps
Feb 25 15:39:55 corecode> good idea
Feb 25 15:40:19 coolvibe> yeah, just default to plain POSIX, and anything extra some OS has, you can add a cap
Feb 25 15:40:51 joerg> corecode: replace -DNOSENDMAIL with -DNOCRYPT, the former is a better example for the use of subpackages
Feb 25 15:40:53 coolvibe> for instance we could have “has_varsyms=1″ or something
Feb 25 15:41:23 corecode> coolvibe: i’d put that into the implementation somewhere
Feb 25 15:41:34 corecode> package description files won’t care about this stuff
Feb 25 15:41:50 corecode> joerg: yea?
Feb 25 15:41:58 coolvibe> or NO_IDEA :)
Feb 25 15:42:17 joerg> corecode: search for it, it’s just the a bad example
Feb 25 15:42:34 joerg> coolvibe: even better :)
Feb 25 15:42:44 corecode> okays
Feb 25 15:42:53 corecode> well, crypto will do
Feb 25 15:43:28 corecode> i should put this stuff on CVS or such
Feb 25 15:45:04 debugger> corecode: “If package flags/flavors
Feb 25 15:45:04 debugger> changed their meaning or got added/deleted it might be desirable for the user
Feb 25 15:45:04 debugger> to get asked to review the settings; ”
Feb 25 15:45:09 debugger> that is so cool :)
Feb 25 15:45:16 corecode> yea it is
Feb 25 15:45:38 debugger> I was thinking on doing that for FBSD hehehe
Feb 25 15:45:46 corecode> now we only need an implementation
Feb 25 15:45:52 debugger> yeah ;)
Feb 25 15:46:04 corecode> subversion got released?
Feb 25 15:46:26 debugger> it seems so. but why svn? use arch :P
Feb 25 15:47:43 corecode> no, i’ll stick with cvs
Feb 25 15:47:52 corecode> no requirements to install on os x
Feb 25 15:48:15 debugger> yeah thats true.
Feb 25 15:49:50 corecode> how do you create a module in cvs?
Feb 25 15:49:54 corecode> always import?
Feb 25 15:50:32 joerg> ?
Feb 25 15:50:38 corecode> like src
Feb 25 15:50:41 corecode> or dfports
Feb 25 15:50:48 corecode> how do i create such directories?
Feb 25 15:51:09 debugger> you can create an “alias” to an existing directory and call it a module.
Feb 25 15:51:16 corecode> nono
Feb 25 15:51:24 corecode> i got a virgin cvs repo
Feb 25 15:51:30 corecode> and want to add a directory
Feb 25 15:51:44 debugger> cvs does not support directories. you have to import a file.
Feb 25 15:51:52 joerg> good question :) never set up a CVS repository
Feb 25 15:51:57 corecode> lol
Feb 25 15:52:16 debugger> you just import files corecode :D
Feb 25 15:52:42 debugger> you have to create a dummy file inside a directory to “import” the directory
Feb 25 15:52:49 joerg> I prefer arch/svn since it allows atomic commits. and arch works with any FTP server w/o any fancy server support
Feb 25 15:53:33 debugger> arch works without borders. that why I like it :D
Feb 25 15:53:43 corecode> how does arch work?
Feb 25 15:54:15 joerg> your repository is just a normal directory and a tarball for each changeset
Feb 25 15:54:27 joerg> and some logfiles
Feb 25 15:55:00 debugger> you can have several repositories distributed by several different “servers”; there is no central repository. also, I can hack on a private copy of a public project and incorporate the changes without to much effort.
Feb 25 15:55:44 debugger> see
Feb 25 15:56:14 joerg> e.g. the “central” repository just gets the changesets via merge of commiters
Feb 25 15:56:55 corecode> ah
Feb 25 15:57:06 corecode> i’m too lazy to learn a new SCS now
Feb 25 15:57:15 corecode> sticking with CVS atm
Feb 25 15:57:51 joerg> corecode: this decision can wait for a while :)
Feb 25 15:58:16 joerg> but CVS is not really a sensible tool for this
Feb 25 15:58:49 debugger> CVS had is time. that time is fading away heheh
Feb 25 16:17:50 coolvibe> cvs works well enough for me
Feb 25 16:19:02 coolvibe> btw, netbsd systinst is pretty weird stuff
Feb 25 16:19:18 coolvibe> it generates scripts which it execs later
Feb 25 16:19:20 debugger> corecode: that thoughts paper is quite complete (and broad!)
Feb 25 16:19:20 joerg> CVS has it uses and it reliable, nobody really said something else
Feb 25 16:19:46 corecode> debugger: yea, been writing on that for some weeks now
Feb 25 16:19:54 corecode> and thinking about stuff for months
Feb 25 16:19:54 debugger> corecode: have you talked with portage guys?
Feb 25 16:20:25 corecode> i want to mail the lists and discuss stuff first
Feb 25 16:20:26 debugger> corecode: yeah, it seems so. quite a think you have there :)
Feb 25 16:20:31 corecode> debugger: nope
Feb 25 16:20:34 debugger> err s,think,thing
Feb 25 16:20:57 corecode> i didn’t talk with any package system guy
Feb 25 16:21:07 corecode> they tend to stick to their own system
Feb 25 16:21:08 corecode> :)
Feb 25 16:21:11 coolvibe> cvs does crunch crunch crunch on my dragonfly source… hm, there could be a song in there
Feb 25 16:21:31 debugger> corecode: you should bring that to them; they might be interested :D
Feb 25 16:21:40 debugger> corecode: oh that I dont knwon :/
Feb 25 16:21:56 corecode> at least that’s my experience
Feb 25 16:22:04 corecode> and portage’s code sucks
Feb 25 16:22:17 coolvibe> corecode: it’s written in a nice language though
Feb 25 16:22:30 debugger> but it seens to work hehehe ;)
Feb 25 16:22:47 coolvibe> debugger: portage sucks when deinstalling dependancies of packages
Feb 25 16:23:02 joerg> codecode: the biggest problem of portage is the missing separation between package management stuff [low requirements] and package building stuff
Feb 25 16:23:09 debugger> yes, Python (or Ruby) is a great choise to implement this.
Feb 25 16:23:19 corecode> if they wrote proper code, yes
Feb 25 16:23:22 corecode> python is nice
Feb 25 16:23:49 coolvibe> well, having a scripting language in the base would be kinda iffy, look at perl for instance
Feb 25 16:24:23 debugger> why in base? it could be a package itself :D
Feb 25 16:24:38 debugger> with a private copy of python too.
Feb 25 16:24:56 coolvibe> debugger: oh, I was going to throw a chicken and egg situation at you
Feb 25 16:25:02 coolvibe> but maybe a very stripped python
Feb 25 16:25:12 JustinS> please oh god no fixed language in base install
Feb 25 16:25:54 coolvibe> maybe OTS can fetch and build python if it can’t run it :)
Feb 25 16:26:00 joerg> JustinS, corecode: that’s what I meant :)
Feb 25 16:26:07 coolvibe> that way we don’t have to pull it in the base
Feb 25 16:26:31 coolvibe> and after it build python, it uses it to record the python package
Feb 25 16:26:34 corecode> read the text to the base
Feb 25 16:26:39 corecode> i’ve mentioned that too
Feb 25 16:26:41 joerg> coolvibe: Python as part of the build system is OK, but not the management tools (installation, deinstallation ,etc)
Feb 25 16:27:10 coolvibe> joerg: yeah, I’m working on the installer now, mostly C and shell stuff
Feb 25 16:27:15 corecode> just want to iron out major mistakes before
Feb 25 16:27:25 corecode> coolvibe: huh?
Feb 25 16:27:32 corecode> you working with rob?
Feb 25 16:27:42 coolvibe> corecode: yeah kinda, we’re competing, as we agreed :)
Feb 25 16:27:49 debugger> joerg: why not? whats the problem with that?
Feb 25 16:27:57 joerg> debugger: ?
Feb 25 16:28:10 debugger> joerg: “but not the management tools (installation, deinstallation ,etc)”
Feb 25 16:28:27 coolvibe> corecode: I’m hacking something together we can use until rob has his stuff ready
Feb 25 16:28:34 corecode> uah
Feb 25 16:28:42 corecode> better combine the effort, no?
Feb 25 16:28:48 coolvibe> corecode: we already are
Feb 25 16:28:49 joerg> debugger: for those stuff no additional requirements should be used, ideally 2 static apps :)
Feb 25 16:29:10 coolvibe> corecode: right now I’m just making the disklabel/fdisk part more ‘friendly’
Feb 25 16:29:13 joerg> debugger: avoids a lot of head aches
Feb 25 16:29:14 corecode> joerg: won’t work
Feb 25 16:29:21 corecode> management is really *big*
Feb 25 16:29:32 corecode> it needs to provide lotsa features
Feb 25 16:29:46 corecode> more than portupgrade does now
Feb 25 16:29:48 joerg> corecode: there are different parts. e.g. consider the apt vs. dpkg split
Feb 25 16:29:59 coolvibe> people can handle cpdup, fdisk and disklabel are somewhat tall orders for some :)
Feb 25 16:30:18 coolvibe> (especially when they want to preserve other operating systems on the disk)
Feb 25 16:30:32 joerg> corecode: I don’t like the idea of providing two views of the build system.
Feb 25 16:31:05 joerg> corecode: if space/inode usage is a problem, you should use a different machines and build packages there
Feb 25 16:31:29 debugger> that I agree :D
Feb 25 16:31:47 corecode> joerg: why not?
Feb 25 16:32:17 corecode> the user just doesn’t need to handle patch files for example
Feb 25 16:32:22 corecode> the maintainer does
Feb 25 16:32:25 joerg> corecode: it just complicates the stuff
Feb 25 16:32:36 corecode> well it’s just a thought
Feb 25 16:32:42 corecode> needn’t be the way to go
Feb 25 16:32:49 joerg> corecode: but often users want to adjust patch files too
Feb 25 16:33:06 joerg> corecode: it just cries for problems :)
Feb 25 16:33:08 coolvibe> or provide their own
Feb 25 16:33:22 joerg> cv: yeah
Feb 25 16:33:24 corecode> joerg: well… i’d say
Feb 25 16:33:33 corecode> MOST will use binary packages
Feb 25 16:33:42 joerg> for those it doesn’t matter
Feb 25 16:33:43 corecode> SOME will build from souce
Feb 25 16:33:59 corecode> and VERY FEW will adjust package descriptions themselves
Feb 25 16:34:00 coolvibe> someone has to build these binary packages
Feb 25 16:34:09 corecode> those really can convert the stuff back
Feb 25 16:34:15 coolvibe> can’t we do some p2p package distribution thing?
Feb 25 16:34:16 corecode> coolvibe: build cluster
Feb 25 16:34:51 joerg> corecode: but it means to either provide a special server app for downloading up2date build descriptions or keeping 2 repositories in sync
Feb 25 16:35:00 coolvibe> if we have crypto signatures and checksums, the package distribution can go through a p2p protocol of some sort
Feb 25 16:35:01 corecode> nonono
Feb 25 16:35:05 corecode> noho!
Feb 25 16:35:11 coolvibe> optionally of course
Feb 25 16:35:12 coolvibe> :)
Feb 25 16:35:17 corecode> repo contains maintainer view
Feb 25 16:35:32 joerg> that what I thought too
Feb 25 16:35:52 corecode> and this can be compiled into enduser view
Feb 25 16:35:53 coolvibe> p2p makes that ‘everyone is a build cluster’ thing happen :)
Feb 25 16:36:11 corecode> let’s keep it realistic :)
Feb 25 16:36:26 coolvibe> corecode: hehe, well, a man can dream, can he? :)
Feb 25 16:36:40 coolvibe> it would be seriously cool stuff if it were possible
Feb 25 16:36:42 joerg> corecode: that’s what I meant. either you need a cron job/dynamic server app/… or have a second repository for the “user” view
Feb 25 16:37:08 corecode> that’s no real repo
Feb 25 16:37:11 coolvibe> it would also provide one other legal use for p2p
Feb 25 16:37:22 corecode> that’s one file or whatever that gets synced by rsync
Feb 25 16:37:24 corecode> dunno
Feb 25 16:38:06 joerg> I don’t think having a few more _inodes_ does matter for someone build from source
Feb 25 16:38:21 corecode> few?
Feb 25 16:38:27 corecode> did you lately count ports?
Feb 25 16:38:28 corecode> :)
Feb 25 16:38:29 joerg> space usage should be
identical [with an efficient filesystem]
Feb 25 16:38:57 joerg> corecode: one description file per port vs. Makefile+patches+descriptions+…
Feb 25 16:39:00 corecode> it isn’t
Feb 25 16:39:25 joerg> with something like reiserfs it would
Feb 25 16:39:26 corecode> files are much below 4k which is the default block size
Feb 25 16:39:37 joerg> that’s what I meant with efficient
Feb 25 16:39:38 coolvibe> well, lotsa files wouldn’t matter much if we had an extent based fs
Feb 25 16:39:38 corecode> 76k files
Feb 25 16:39:40 corecode> in ports
Feb 25 16:40:08 coolvibe> corecode: what about some read-only tarfile filesystem of some sorts?
Feb 25 16:40:15 corecode> vs 10k ports
Feb 25 16:40:21 coolvibe> then only one file is used, but in the container there will be many
Feb 25 16:40:33 corecode> yea, or one fat xml file :)
Feb 25 16:40:38 coolvibe> eek
Feb 25 16:40:42 corecode> hm?
Feb 25 16:40:47 joerg> also having static description i.e. of the dependencies will be difficult for things like flavours/subpackages
Feb 25 16:41:01 joerg> don’t use XML for such a thing
Feb 25 16:41:07 corecode> why not?
Feb 25 16:41:07 coolvibe> yah
Feb 25 16:41:16 joerg> corecode: inefficient parsing
Feb 25 16:41:23 corecode> okay, what then?
Feb 25 16:41:25 corecode> make?
Feb 25 16:41:26 corecode> sh?
Feb 25 16:41:29 corecode> tcl?
Feb 25 16:41:36 coolvibe> sqlite!
Feb 25 16:41:36 coolvibe> :)
Feb 25 16:41:45 corecode> proprietary format
Feb 25 16:41:46 corecode> ?
Feb 25 16:41:47 JustinS> modula-3
Feb 25 16:41:47 coolvibe> no
Feb 25 16:41:52 JustinS> :)
Feb 25 16:41:54 coolvibe> sqlite is public domain
Feb 25 16:42:00 joerg> eek
Feb 25 16:42:03 coolvibe>
Feb 25 16:42:18 joerg> we have no PD here :(
Feb 25 16:42:27 corecode> wasn’t targeted at sqlite
Feb 25 16:42:30 joerg> corecode: documented hierachical file format
Feb 25 16:42:59 coolvibe> joerg: the sqlite license is very very relaxed (almost nonexistant)
Feb 25 16:43:17 joerg> coolvibe: as long as it has a license, everything is alright
Feb 25 16:43:35 coolvibe> joerg: it also has many language bindings
Feb 25 16:43:53 coolvibe> joerg: that way we can have the flexibility of an RDBMS without having to install one
Feb 25 16:44:04 joerg> coolvibe: I know of the language bindings
Feb 25 16:44:04 coolvibe> it’s pretty fast too
Feb 25 16:44:47 coolvibe> maybe we can use that instead of XML
Feb 25 16:46:24 joerg> well, as backend for the package management it might be useful
Feb 25 16:46:26 coolvibe> it also has no real external dependancies
Feb 25 16:46:32 coolvibe> indeed
Feb 25 16:47:03 corecode> yea
Feb 25 16:47:27 corecode> i’d use some format so that different frontend programs can use the database
Feb 25 16:47:51 coolvibe> corecode: with sqlite, they just link in the lib, and fire off sql queries
Feb 25 16:47:59 corecode> well
Feb 25 16:48:12 corecode> then they all need to implement stuff from scratch
Feb 25 16:48:23 corecode> we need a package API
Feb 25 16:48:28 coolvibe> it’ll be a lot better than /var/db/pkg
Feb 25 16:48:45 coolvibe> and for safety, periodic dumps can be made
Feb 25 16:48:49 corecode> which in turn needs to be written in some language C can be linked with
Feb 25 16:48:56 joerg> coolvibe: /var/db/pkg/{one file for each package} + /var/db/pkg/master.db
Feb 25 16:49:05 coolvibe> joerg: a.k.a lots of files
Feb 25 16:49:25 joerg> coolvibe: come on, you don’t have tousands of packages
Feb 25 16:49:39 coolvibe> I have none at the moment ;P
Feb 25 16:49:42 joerg> and if you have, that doesn’t matter
Feb 25 16:49:49 corecode> joerg: but ports is at 10k
Feb 25 16:50:02 corecode> and this needs to be handled
Feb 25 16:50:02 coolvibe> yeah, what if some yahoo does make all install in /usr/ports
Feb 25 16:50:14 joerg> corecode: for /var/db/pkg you have only the installed packages
Feb 25 16:50:24 corecode> yes
Feb 25 16:50:26 corecode> that
Feb 25 16:50:41 coolvibe> the /usr/ports root as a makefile
Feb 25 16:50:52 coolvibe> so, potentially, _every_ package can get installed
Feb 25 16:51:01 joerg> and I want to keep an independent version beside the database for recovery purposes
Feb 25 16:51:12 coolvibe> joerg: dump it periodically
Feb 25 16:51:14 corecode> joerg: backup database?
Feb 25 16:51:21 coolvibe> corecode: sql dump :)
Feb 25 16:51:25 coolvibe> which is editable
Feb 25 16:51:34 corecode> it should be edible
Feb 25 16:51:48 coolvibe> edit sql dump and reinit the db
Feb 25 16:52:09 corecode> question is: do we need relational data structuring for that?
Feb 25 16:52:15 joerg> corecode: adding a file and syncing is safer
Feb 25 16:52:32 coolvibe> corecode: it could help in some places, it would for dependancies and searches
Feb 25 16:52:34 joerg> corecode: it is useful for querying the stuff, otherwise using db(3) is enough
Feb 25 16:52:49 coolvibe> joerg: db is only key->value, which is kinda limiting
Feb 25 16:52:52 corecode> coolvibe: requirements, yea, partly
Feb 25 16:53:08 corecode> the dependancy graph still needs to be constructed in memory
Feb 25 16:53:13 joerg> coolvibe: I know, but for most of it, that’s actually enough
Feb 25 16:53:26 joerg> corecode: not for installation/deinstallation
Feb 25 16:53:42 corecode> no, not for that
Feb 25 16:53:56 corecode> but for requirements resolve before
Feb 25 16:53:57 coolvibe> well, impact of deinstallation might be considered
Feb 25 16:54:11 coolvibe> what stuff will break if you deinstall package xyz
Feb 25 16:54:47 joerg> coolvibe: do we need anything above recursion deletes?
Feb 25 16:55:08 joerg> corecode: no, you don’t need a full dependency tree for that
Feb 25 16:55:17 corecode> deinstall takes place after conflict resolvement
Feb 25 16:55:51 coolvibe> joerg: well, being able to dry-run certain actions (to see what would happen) would be nice
Feb 25 16:56:15 joerg> I can think only of three basic actions: package installation, deinstallation and replacement
Feb 25 16:56:29 joerg> the first needs to add additional packages
Feb 25 16:56:38 joerg> the second to delete recursive
Feb 25 16:56:51 joerg> the third only add deps too
Feb 25 16:57:04 corecode> uhm
Feb 25 16:57:35 joerg> I want to cleanly separate the low level package management from high level operation like “emerge world”
Feb 25 16:57:43 coolvibe> joerg: oh, like in my case with netbsd yesterday/today, where one lib (say libxml) caused all of kde to recompile? no thanks :)
Feb 25 16:58:27 coolvibe> if you replace a package, you could hang on to the old version
Feb 25 16:58:32 coolvibe> so older apps don’t break
Feb 25 16:58:48 coolvibe> multiple versions need to be managed too, as outlined in corecode’s doc
Feb 25 16:59:06 coolvibe> so it would also be handy to see what packages still use olde libs
Feb 25 16:59:28 coolvibe> and if nothing is connected to the older libs/apps, they could be safely removed
Feb 25 17:00:23 corecode> yea
Feb 25 17:00:31 coolvibe> hm, I’m getting sigills
Feb 25 17:01:13 joerg> after some more thinking, even resolving of dependencies should _not_ happen in the low-level tools
Feb 25 17:01:37 coolvibe> joerg: yeah, but you should be able to interrogate the system
Feb 25 17:01:43 joerg> coolvibe: handling of multiple versions is a requirement per-se
Feb 25 17:01:59 coolvibe> a database that can do more than store key->value would be nice here
Feb 25 17:02:05 corecode> but let’s postpone implementation discussion
Feb 25 17:02:15 coolvibe> the simple tools only have to do simple inserts and queries
Feb 25 17:02:32 corecode> joerg: right. low level tools could just be tar / xargs rm :)
Feb 25 17:02:57 joerg> corecode: that’s the point
Feb 25 17:03:17 corecode> or some other programs equally intelligent
Feb 25 17:03:31 coolvibe> ah, and a higher tools that do package registering
Feb 25 17:03:39 coolvibe> installation is simple, yeah
Feb 25 17:03:40 joerg> I think we will end up with the following: {install, deinstall, replace} package, update db, rebuild db
Feb 25 17:03:48 corecode> but, for the user the differenciation shouldn’t be visible
Feb 25 17:04:03 coolvibe> also, I don’t think binary updates will work on source built packages, so we need to flag those
Feb 25 17:04:11 joerg> yes, that should be done by something like apt
Feb 25 17:04:38 corecode> coolvibe: huh?
Feb 25 17:04:38 joerg> coolvibe: you can record the origin of the package together with a UUID
Feb 25 17:04:43 coolvibe> corecode: bsdiff
Feb 25 17:04:44 corecode> coolvibe: right
Feb 25 17:04:45 corecode> :)
Feb 25 17:05:01 joerg> coolvibe: and patches update from one UUID to another
Feb 25 17:05:11 coolvibe> oh yeah, that would be cool
Feb 25 17:05:19 joerg> me too :)
Feb 25 17:05:23 corecode> :D

Posted by     Categories: Goings-on     1 Comment
1 Comment on Packaging discussion log


  1. PCN says:

    Heheh. These guys are getting nowhere fast. Not that I have any brilliant ideas either. I think that this simple issue is going to be a big pain in the butt to get going, as it’s not quite as clear as the project’s other goals.