poiu
poiu is an asdf extension that builds in parallel.

To use it, just start your build scripts with

(require :asdf) (asdf:load-system :asdf) (asdf:load-system :poiu)
i.e. add the latter clause, for you should already have the former clauses (and possibly some asdf configuration between the require and the first load-system). Make sure that you're using a recent ASDF 3 together with a recent POIU. POIU will modify the asdf::*default-plan-class* so subsequent calls to asdf:load-system will implicitly use it.

You can find POIU on the QITAB homepage: http://common-lisp.net/project/qitab

POIU will notably compile each Lisp file in its own forked process, in parallel with other operations (compilation or loading). However, it will load FASLs serially as they become available. In deterministic mode (the default), it will load files in a deterministic order based on the state at the start of the build (usually, from clean). If calling operate with :deterministic-p nil, it will load files opportunistically in an order that depends on which compilation finishes first, sacrificing determinism to a further parallel speed up.

POIU will only make a difference with respect to ASDF if the dependencies are not serial (i.e. no difference for systems using :serial t everywhere). You can however use Andreas Fuchs's ASDF-DEPENDENCY-GROVEL to autodetect minimal dependencies from an ASDF system (or a set of multiple such).

POIU may speed up compilation by utilizing all CPUs of an SMP machine. POIU may also reduce the memory pressure on the main (loading) process. POIU will enforce separation between compile- and load- time environments, helping you detect when :LOAD-TOPLEVEL is missing in EVAL-WHEN's (as needed for incremental compilation even with vanilla ASDF). POIU will also catch *some* missing dependencies as exist between the files that it will happen to compile in parallel (but won't catch other dependencies that may otherwise be missing from your system).

When a compilation fails in a parallel process, POIU will retry compiling in the main (loading) process so you get the usual ASDF error behavior, with a chance to debug the issue and restart the operation.

POIU was currently only made to work with SBCL, CCL, CLISP and Allegro. Porting to another Unix Lisp implementation that supports ASDF should not be difficult.

Warning to CCL users: you need to save a CCL image that doesn't start threads at startup in order to use POIU (or anything that uses fork). Watch QITAB for some package that does just that, single-threaded-ccl.

To use POIU, just load poiu.lisp after asdf.lisp is loaded, then use it with (asdf:load-system :your-system) as usual. The asdf::*default-plan-class* will have been modified to use POIU. Once again, you may want to first use asdf-dependency-grovel to minimize the dependencies in your system.

POIU was initially written for SBCL by Andreas Fuchs in 2006 as part of an experiment funded by ITA Software, Inc. It was later modified by François-René Rideau, who wrote the ports to CCL, CLISP, Allegro, maintained POIU as ASDF became ASDF 2 and further evolved, and completely rewrote it for ASDF 3. The original copyright of ASDF (below) applies to POIU.