A minor update fixing the handling of path names with spaces as well as delayed vendor branches.
A minor update fixing the handling of path names with spaces as well as delayed vendor branches.
The new snapshot of cvs2fossil fixes two issues seen in the NetBSD repositories:
Both changes have been in use for the NetBSD conversions for a while.
Checkout https://github.com/jsonn/src and https://github.com/jsonn/pkgsrc!
Checkout https://github.com/jsonn/src and https://github.com/jsonn/pkgsrc!
After some initial issues the git conversion seems to be stable now. Both src and https://github.com/jsonn/pkgsrc can be found on github.
Fun fact about git: pushing trunk from src alone and the other refs with push -all afterwards requires 50% more space than doing the push -all in first place.
One issue remains. I can't do incremental exports from fossil to git, because git can't find objects it has imported earlier. I'm not the first person to experience this, but it seems no one cares enough to fix it.
New server with better connection for the fossil repositories, a mirror and initial addition of git support
New server with better connection for the fossil repositories, a mirror and initial addition of git support
The fossil repositories (src and pkgsrc) have moved to a new server, so at least upstream bandwidth no longer is a problem. For cloning, it is still recommented to fetch the database files directly from ftp.NetBSD.org. Zafer Aydoğan also provides mirrors at src and pkgsrc.
The src repository has been rebuild from scratch after many clean ups to the branching. It should be in a pretty good shape now. Special thanks to S.P. Zeidler for the assistance in messing up cvs.NetBSD.org.
At this point, I believe the conversion to be stable and don't plan any more repository changes. If you run across inconsistencies in the conversion, please mail me though.
I've also started to integrate the git export. This is still a lot more overhead than necessary. The incremental fast-import in git doesn't work, so it has to write all all blobs and commits on every pass, adding another 20min or so to the conversion rounds. The result can be found on github. I'm interested in feedback here to decide asking github for more space to add src as well.
Update: there seems to be a small bug with the conversion of file deletes, so the git repo will be nuked and rebuild soonish
Update 2: fixed and repushed
Bug fix release for cvs2fossil to get stable conversion for pkgsrc
Bug fix release for cvs2fossil to get stable conversion for pkgsrc
The new snapshot of cvs2fossil fixes three important issues:
I've used this and also changed the conversion to create the newer sub-second timestamp format. To fully exploit the bug fixes, the two NetBSD repositories (pkgsrc and src) have been recreated some scratch. I've decided to skip the top-level directory now as well, so src/Makefile is now just Makefile and src/bin/cat/cat.c can be found in bin/cat/cat.c.
In the two month since the last update, the Fossil conversion utility has seen quite a number of improvements.
A public mirror of the repository conversion is provided.
In the two month since the last update, the Fossil conversion utility has seen quite a number of improvements.
A public mirror of the repository conversion is provided.
Since the initial release of the Fossil converter, a lot has changed:
The code can be found here.
The processing time is still around 5h for src, but I'm currently running the machine with less memory. On my laptop with SSD, it needs 3h for a full conversion.
I am providing a public mirror of the conversion. It is updated around 3-4 times a day. Please avoid cloning directly from that site as I am a bit bandwidth constrained. You can fetch the repository directly from ftp.netbsd.org (which is much faster) and pull changes from my server afterwards. Please also note that the leaf versions are sometimes not completely stable due to incomplete rsync of large commits or if a branch was created without a commit between runs. When I see such an instance, I will close the wrong leaf.
What is left before USE_DESTDIR=yes can be the default for everyone?
What is left before USE_DESTDIR=yes can be the default for everyone?
DESTDIR support is one of the biggest internal changes since the introduction of package barrier and it has some user visible consequences.
What is left to do before it can be made the default to keep the impact as small as possible?
A new name for a build phase, a new behavior for DESTDIR users and some black magic.
A new name for a build phase, a new behavior for DESTDIR users and some black magic.
When the DESTDIR support was originally introduced 3.5 years ago, I tried to keep it as non-intrusive in the infrastructure as possible.
The reason was simply, that it is very easy to introduce bugs in the complex make rules and DESTDIR support would need quite a bit of work before it is ready for main stream consumption. My initial goal for 2006Q4 was 50% or so I wrote in my EuroBSDCon slides. I was waaaay too optimistic.
Fast forward 3 years to the current time. DESTDIR support is now supported by almost all packages. 384 package locations without DESTDIR support remains (compared to 8302 with). To avoid unnecessary regressions from developers not testing it, I made USE_DESTDIR=yes the default for PKG_DEVELOPER a while ago. That's when the complains and insults started.
Due to the change, "make install" now suddenly doesn't modify /usr/pkg any longer. The other direct consequence is that "DEPENDS_TARGET=package" breaks as well.
The former is reasonable to fix by finally pushing the originally planed intrusive changes. The latter is more difficult to do without second guessing and more a case of Update your config, please. So, what was needed to change the install target to behave the same from a user perspective?
First of all, a bit of search and replace. Second, dealing with the surprises. Just renaming install to stage-install -- fails. The install-vars target has be renamed as well, as the necessary files are created. More fun was the addition of the new target. Just adding
.PHONY: install .if ${_USE_DESTDIR} == "no" install: stage-install .else install: package-install .endif
...doesn't work. It surprisingly does nothing. What happened?
pkgsrc computes a number of variables based on the dependencies. For example, BUILDLINK_PREFIX depends on whether the package is builtin and if not, where it is found. This got quite expensive when pkgviews was introduced. A long time ago, pkgsrc therefore run a new make instance for every major build phase (depends, extract, build, install). This was changed later with the introduction of the pkgsrc barrier. Essentially, the build phases are separated into pre-barrier and post-barrier phases, with a separate make invocation in between.
How does this affect the install target above? install has to be a post-barrier operation otherwise it ends up effectively invocing build, but not the desired install rule.
Black magic. Why do I know? The same problem occured with package-install a long time ago...
pbulk includes logic to avoid rebuilding packages that haven't changed based on the RCS ID ($NetBSD$).
A look at some problem cases.
pbulk includes logic to avoid rebuilding packages that haven't changed based on the RCS ID ($NetBSD$).
A look at some problem cases.
pbulk contains a script to determine whether a package has to be rebuilt or not. There are three checks performed by default:
The third condition is automatically valid after a bulk build, otherwise the system has time keeping or file system issues. The other conditions are more interesting.
The first condition can trigger a permanent rebuild if files are formed in a way that the +BUILD_INFO processor extracts incomplete or additional RCS IDs. The rdigest package for example had a patch starting with
@@ -41,8 +41,17 @@ __RCSID("$NetBSD: digest.c,v 1.15 2007/0
As the RCS IDs are extracted with a simple grep expression, this ended up overwriting the original RCS ID at the top.
Another example was url2pg, which contained the following statement in a local file:
print PLIST ("\@comment \$NetBSD\$\n");
This is picked up too and should be escape, in this case by splitting the string into two parts in the middle of NetBSD.
Interestingly, the second condition is violated in one case as well. p5-DBIx-Class-EncodedColumn depends on p5-Digest-SHA. The version of p5-Digest-SHA is higher than the version of the Perl package, so the dependency resolver picks up that. During the build, pkgsrc decides that perl is already present and good enough, so skips the dependency. I'm not sure how to best address this yet.
After the DESTDIR changes, emulators/handy_sdl started to fail in the bulk builds, even though it works fine when build by hand. A failure analysis.
After the DESTDIR changes, emulators/handy_sdl started to fail in the bulk builds, even though it works fine when build by hand. A failure analysis.
The bulk builds are a great way to find build issues. Sometimes they are pretty puzzling though, because they show issues that don't appear otherwise.
I recently added the full DESTDIR support for emulators/handy_sdl, after making sure that the do-install rules are fine. The package compiled fine during my tests, but failed in the next run of pbulk with a dependency error from gmake. This was very puzzling as it was still working when starting the build manually.
The first idea was to look for the same problem that broke some Emacs packages. The Emacs packages had been failing in the bulk build for the obvious reason with a rather impossible error message (-batch missing when it was clearly specified). After some digging it turned out that the Makefile used the variable BATCH, which is already set in bulk builds to prevent interactions e.g. when patches don't apply. Turned out to be a dead end.
After some scratching the head I tried to fully replicate the build and started it as unprivileged user. As pkgsrc allows building packages with PKG_DESTDIR_SUPPORT=user-destdir without privileges, pbulk naturally uses that too. That was an instant hit -- the build fails if done unprivileged. Looking at the tarball of the work area from the bulk build, it became obvious why. The src directory is not executable and therefore gmake couldn't scan it. Turns out, a single chmod -R a+x fixes the problem.
20 pages of changes on http://pkgsrc.se:
20 pages of changes on http://pkgsrc.se:
To welcome the new year, I wanted to get rid of some of the ancient cruft. pkgsrc like many other systems tends to accumulate unmaintained software. In many cases, this software tends to be broken in the bulk builds, has special restrictions on redistribution or is a security nightmare.
For the post-2009Q1 cleanup, the basic criterions were:
Sadly, removing PHP completely is not an option, but it has been over a year since the final EOL of PHP 4 and therefore enough time to migrate the remaining code to a supported code base.
Java has similar issues. The Linux emulation for the older Sun JDK/JRE releases was often problematic. The licensing doesn't really help either, e.g. it is virtually impossible to build and distribute a JDK 1.5 as native code. With the advent of GPLed OpenJDK 7 the situation has improved dramatically.
PKG_DEVELOPER now implies USE_DESTDIR=yes by default.
This helps to catch bugs and simplifies updates.
PKG_DEVELOPER now implies USE_DESTDIR=yes by default.
This helps to catch bugs and simplifies updates.
After a long time working in the background, I have started the process of making USE_DESTDIR=yes the default behavior.
The first step was making developers use it by default.
This has a number of implications:
After a small round of changes with the help of tnn, "make replace" now fully works with USE_DESTDIR as well.
Before, it couldn't replace identical versions. The new pkg_add -U option explicitly handles that case.
What next? There are still slightly over 600 packages that don't support USE_DESTDIR.
I hope to further reduce this to 400 or so, at which point the current warning will turn into an error.
I plan to make USE_DESTDIR the default soon for all useres, "make replace" was the last major blocker.
Hopefully, in the second half of 2010 it will be required and support for USE_DESTDIR=no can be removed.