If you didn't live through Operating Systems of the 70s, 80s, and 90s (or caught the lecture at university), then you might want to read this ArsTechnica article on the history of the more mainstream filesystems so you can fake your way authoritatively when the topic comes up.

(It doesn't cover less mainstream or specific-purpose filesystems or cover more interesting work done on some of the filesystems it does mention - particularly on the BSD side of things.  You'll have to do your own research on those.  Try FreeBSD 7.0, maybe?)

FreeBSD 7.0

I'm a bit behind the times on this one (Tectonic had a story immediately), but FreeBSD 7.0 was announced two weeks back.  For more information, check out this interview with a few FreeBSD developers about FreeBSD 7.0 as well as this FreeBSD 7.0 preview presentation that contains quite a bit on performance improvements.

How cool is this - a South African (Capetonian, at that) student, Federico Lorenzi, placed in the top 10 of the first Google Highly Open Participation Contest.  Federico has good taste - he likes FreeBSD and Python - and worked on the MoinMoin wiki project for the contest.

The Google Highly Open Participation Contest is a way to introduce pre-university students to the world of open source, and to reward the contributions made by these students.  The winners get flown over to Google's Mountain View headquarters for a prize-giving.

In conjunction with Charl van Niekerk's Google Summer of Code participation, the fact that South Africa has a wealth of technical talent is probably reaching many more ears.  (I'm sure Google isn't surprised - I know a few South Africans doing good work there.)

The Cape Town Linux User Group was lucky to get both a behind-the-scenes and front-end explanation by Johan Hartzenburg of ZFS - Sun Microsystems' new advanced all-singing all-dancing filesystem which is also a volume manager and, I'm sure will eventually be able to send email before becoming Emacs.

Johan explained to us how ZFS manages to always be consistent - by never editing existing metadata entries, but rather copying the entry to a new entry, editing the new entry, and then replacing the link of the original entry's parent to the new entry.  But, of course, because it never edits an entry directly, the parent goes through the same process, until it reaches the uberblock.  The uberblock never has a new copy created of itself, but there are multiple copies of it, and updating the uberblock is an atomic operation.  Even if things go awry while this is half-complete, any of the uberblocks is consistent (and, I think it has a timestamp to fall back on).

This all sounds really inefficient, but ends up not being so.  The new blocks are generally all written near each other, making a whole bunch of random writes actually often be more efficient by having all the new data and metadata all be written near each other.

Unused metadata and data blocks are then removed.

Using this design makes snapshots pretty trivial - since all you need to do is not delete the original metadata and data blocks used in the snapshot.  Everything speeds on ahead, and the scrubber just doesn't free those blocks.

Also, using this design makes changing on-disk options pretty simple.  This includes, for example, how ZFS can efficiently handle different byte orders.  On read, ZFS can handle either order, but on write will always use the most efficient byte order.  Similarly, compression can be used on a data block level - every time a change happens to a file, it can compress the new data block that is created.

This also includes how to bring more members into the pool and harness the increased I/O bandwidth.  The "allocator" just needs to allocate the new blocks created to be edited to the new member, and as time goes by, all members of the pool naturally tend to have equal amounts of data, and thus maximising bandwidth in concurrent read or write requests.

The command line tools are incredibly simple and powerful, and with ZFS you don't have to worry about device renaming, as it records on the disks all the information necessary to find out where in the ZFS hierarchy that disk lives. Easy to use, and hard to screw up?  How can it possibly succeed?

Solaris servers with Xen as Dom0 (which seems to be progressing well) with a massive ZFS storage pool and multiple virtual machines just so sounds like a winning plan.  Or FreeBSD, once ZFS-on-FreeBSD (going well, I see) and Xen-Dom0-on-FreeBSD (not quite as encouraging) are available in stable forms.

Andre Oppermann stepped up to the plate to overhaul and clean up the TCP/IP stack in FreeBSD, and I was one of his funders. Well, I was part of a funding contribution by a group of South African FreeBSD users and friends that raised R3.5k (440 euro). It gives me an evil plan, except I need R20k. More on that later, perhaps.
Tags: ,
Just noticed that Arved has a list of FreeBSD committers that have web logs. I wonder if there's a Planet FreeBSD...
Tags: , ,
Well, after importing 850k new articles into our system, we started experiencing more frequent problems with MySQL. Well, I tracked this down to a FreeBSD 4.x kernel limit on the maximum data size of a process. MAXDSIZ is the thing to tune to fix this.
Tags: , ,
As part of the migration (mentioned twice previously, I need to replace vpopmail's (increasingly awkward) ability to do POP-before-SMTP with courier-imap. DRAC was the answer.
Tags: ,
I'm setting up a new mail server, and wanted to get the on-board Serial ATA RAID going. Adventure inside.
Tags: ,
I've been playing with Debian a bit at home, mostly to find out if it's a usable desktop OS to give to my developers (who really needn't suffer with building from ports and other FreeBSD'isms). Here're my two main irritations.