Data compression and the growing number of cores in modern CPUs

Since very early in my geek life, I've been fascinated by parallel processing and, generally speaking, systems that make use more than one processor. So you can imagine my delight when the era of multicore processors came.

Not exactly so. A frustratingly high number of tools and operating systems still does not use the potential of multiple cores, even in the server world. Virtually all modern computers have at least two of the little guys, yet it's still difficult to keep them busy.

In my dwelling on the internets, in found that the bzip2 compression utility have finally a multi process capable version: pbzip2 (out since a while : first non-beta was released in march 2007). It works great and I started to use it in production with our backup and service scripts. You can either let it detect the number of CPUs, or specify it yourself (and keep some processing power for other tasks) with the -p option - though I suppose the performance increase should be lilmited by disk throughput

I've been able to use it successfully on debian/ubuntu and Macos X. Enjoy.