The FreeBSD Diary

The Billionium came and went and few people noticed. But all cvsup mirrors noticed. And so did some cvsup clients. On Sun Sep 9 01:46:40 UTC 2001, the Unix date changed from 999,999,999 to 1,000,000,000. This uncovered a long-standing bug in CVSup. The bug places a heavy load on the servers not to mention the clients. Upgrading is strongly recommended. All versions of CVSup prior to SNAP_16_1d contain a bug which affects the timestamps of files which have been modified on or after Sun Sep 9 01:46:40 UTC 2001.

If you don't know that 4.4-RELEASE came out, you're probably not on the Net much. I upgraded a few boxes recently. Some were 4.2, the rest were 4.3. The first box I upgraded was in New Zealand. It's actually cvsup.nz.freebsd.org, but it needed to be upgraded in order to install the cvsup bug fix. If you haven't upgraded your local copy of cvsup, you'll soon find yourself unable to any cvsup server. This version of cvsupd will reject clients created prior to the bug fix date. You need to be on at least the following versions:

$ cvsup -v
CVSup client, non-GUI version
Copyright 1996-2001 John D. Polstra
Software version: SNAP_16_1d
Protocol version: 16.1
Operating system: FreeBSD4
http://www.polstra.com/projects/freeware/CVSup/
Report problems to cvsup-bugs@polstra.com

$ /usr/local/sbin/cvsupd -v
CVSup server
Copyright 1996-2001 John D. Polstra
Software version: SNAP_16_1e
Protocol version: 17.0
Operating system: FreeBSD4
http://www.polstra.com/projects/freeware/CVSup/
Report problems to cvsup-bugs@polstra.com

You might think it odd that I choose to first upgrade a box an ocean and a hemisphere away, but it was the one with the most need. The box was on 4.2 and I was unable to install CVSup from source. The packages were for 4.4. Upgrading seems to be the right thing to do. Everything went well. The box came back after I installed the new kernel. And then again after I intalled the new world. Happily, everything about the upgrade went smoothly. That was good.

My next upgrade (of the dual XEON box) was not so successful. After the reboot, the box came back. But I was locked out. I could not get in via ssh. Luckily, this box was only two feet away from me. Granted, the handbook does recommend single user mode.... I suspected the problem was related to my firewall, IP Filter. I don't blame ipf. I blame me. The problem was that ipf was non-functional; it wouldn't load the rules. I suspect this was because I was using a 4.4 kernel with 4.3 world. If I tried to load the rules, something like this was the result:

1:ioctl(add/insert rule): No such process

This is not a good thing....

Not being able to load your firewall rules is a big deal. I logged into the console and modified my kernel configuration file to comment out this option:

options IPFILTER_DEFAULT_BLOCK

This option makes ipf a "block by default" filter. Without that option, ipf is accept by default; if you have no rule set, it will be the same as "pass all from any to any". With that option, ipf blocks everything if you have no rule set.

Under normal circumstances, I recommend using this kernel option if you are using ipf. But this is an exceptional circumstance albeit temporaray. Granted, I could have avoided the problem by not rebooting with the new kernel before installing buildworld. I won't be doing that for the next few boxes, some of which are remote. I solved the problem because I had access to the console. I modified my kernel configuration file to comment out that option, compiled a new kernel, then I rebooted. That gave me ssh access. I returned to my desk, did the install world, and rebooted. Then I again rebuilt my kernel, this time with the above option active, and then rebooted again. That gave me the results I needed.

As a result of the above learning experience, I modified the instructions for my build world script to add some words of caution where ipf is concerned.

Later this week, I then moved onto my other remote box. One with did use ipf, unlike the remote upgrade first mentioned in this article.

I was able to upgrade my firewall by mounting the /usr/obj/ and /usr/src/ directories from the XEON box via NFS. I wrote about that process back in February. I still think it's the fastest way to upgrade multiple boxes. Build once; install many.

I have two more remote boxes to upgrade. One is in New Zealand, the other is in downtown Ottawa (about 25 minutes away by bicycle). Both run ipf. The first box was the colocated webserver. That upgrade went smoothly. No problem. In fact, you're reading pages served up by that box.

The other box in NZ is now installing world. We'll see how that goes. But given that the update for the colo box went well, everything should be fine.