If you haven’t checked out distcc do so forthwith. It’s a distributed build tool that sits on top of make and makes C++ builds significantly faster. Our Linux build previously took 23 minutes on a dual-processor Linux blade; with distcc on 19 hosts, it now builds in under 2:30. Clearly not a linear improvement, but still very impressive.
What I liked most about distcc was how easy it was to install. I literally unpacked it, ran the standard =./configure; make; make install=, set the required environment variable, started daemons on each engine (a 5-line perl script) and ran my build. Amazing! It helped that all our boxes are on a common mount point, so distcc didn’t need to be built on each machine. distcc uses make’s -j flag to distribute as many parallel compilation jobs as you specify and distribute them to as many machine as are available. If it isn’t able to distribute a job, it will run it locally. Output is piped back to the host that is driving the build (this means if you’ve got heavily formatted output, you might see it out-of-order).
We’ve also got Xoreax Incredibuild set up now, for our Windows builds — again, builds that used to take 25-30 minutes previously (we have an extra build step on Windows that slows things down) now take roughly 2:30. Incredibuild was similarly easy to set up — easier, in fact, since it ties in tightly with Visual Studio and all but sets itself up. It’s completely transparent, so there are none of the phantom-network-failure problems that you often get with distributed apps. The other beautiful aspect of Incredibuild is that it runs jobs in low-priority threads, so the performance impact on users’ desktops is minimal. I can honestly say I wouldn’t know my machine was being used for other people’s builds if it wasn’t for the tray icon!
It’s important to note that both distcc and Incredibuild distribute the compile portion of the build, not the linking part of the build (though with VS 2005, Incredibuild apparently lets you do that too!), so after you’ve got all your C++ source compiled, you still spend 30 seconds or so linking everything into a library or executable.
I also don’t fully understand why neither app leverages network-attached-storage (NAS) and instead insist on shipping source between engines themselves. I asked this question on the distcc list and was told that, counter-intuitively, NAS doesn’t scale well for distributed builds.
0 Responses to “distcc pwns (and so does Incredibuild)”