Speeding up build on autofoo projects

First of all, a little digression about build systems.

I’d like to clarify that I’m no lover of autotools. But I’m no lover of any other build system, neither, and autotools is used on several open source projects, including most of the ones I participate. It’s easily copied and pasted by new projects. It’s known by distro maintainers how to hack on it to work on different scenarios like cross-compilation, distribute build across machines, different compilers, rpath, etc etc. Moreover, from my experience, project maintainers usually dislike changing the build system because it causes several headaches not related to the raison d’être of the project. So in general I prefer to modernize the build system rather than radically change it to another one.

Enlightenment window manager is about to be released and I was really bored by the amount it was taking to compile. I use icecream to spread the build across the machines on the local network, so I usually do things like “make -j40″. No matter how many jobs I could put there to parallelize the build it was still painfully slow, particularly while compiling the modules. Taking a look in the Makefile.am files, my suspicion was true: it was using recursive makefiles and since each module has a few source files (circa 3 ~ 6), the build was parallelized only among the files in the same directory. This is because the build is serialized to build each directory. There are plenty of links on the web about why projects should use non-recursive automake. I will enlist some:

So I decided to convert E17′s automake files to non-recursive ones. At least the modules part. After hours of repetitive tasks for converting it, fixing bugs, building out-of-tree, in-tree, distcheck, etc, I committed it and the build time was improved like below:

BeforeAfter
autogen.sh + configure0m47.6s0m36.2s
make -j313m1.9s0m49s
make -j31 with dirty modules only2m38s0m28.2s

So, after configuring it we can build E17 in roughly 1/4 of the previous time.

After the commit introducing the change there were several others to improve it even more, prettify the output, fix some other bugs. It also got reverted once due to causing problems to other developers, but in the end it was applied back.  The worst bug I found was related to subdir-objects option to Automake and Gettext’s Automake macros. That option means that the objects built are kept in the same directory as the correspondent source file. This is needed, particularly in a non-recursive automake scenario, so the objects from different modules don’t conflict due to being put in the same directory.  However, letting this option in configure.ac made “make distcheck” fail in some obscure ways and I later tracked it down to be gettext’s fault. A simple “fix” was to remove it from configure.ac and set it in the “AUTOMAKE_OPTIONS” variable of the modules’ Makefile.am. I really hope someone has the time and will to fix gettext macros – they are a horrible mess and I don’t want to play with them.

6 thoughts on “Speeding up build on autofoo projects”

    1. Yep. There are tons of alternatives. I myself already worked on porting autotools to cmake (for webkit).

      I already played with plenty of them. Every build system has its pros and cons. The biggest pro of autotools is being an already established one with features/hacks for almost all build scenarios one can think of. And people do need the weird scenarios when building for their weird distro/configuration. So, shipping a shiny new build system, whether cool and probably faster, will make package maintainers angry because they cannot use what they already know and have macros, tricks and what not to make it work on their build machines.
      Ask OE folks for instance.

  1. This is a nice and not hyperbolic coverage of the reasons to use non-recursive automake. :) This has come up on some projects I’m involved in, so I feel inclined to point out some drawbacks of this approach.

    1. Requires you to namespace the build much more than previously. Every time a generic variable was used in a recursive Makefile, it now has to be carefully namespaced considering both its effects on the target as well as the automake environment itself (e.g., target_CFLAGS vs AM_CFLAGS). This can be pretty irritating for a large project.

    2. Requires you to adjust the make prerequisites to keep the correct serialization. This can be quite complex for some people. A big advantage of the recursive system are the synchronizing points you’re trying to avoid. In a standard automake system it’s easy to order the build by just putting things that logically depend on another component of the build in a separate subdirectory. Not a big deal for a small project, but can certainly require a lot more thought than previously.

    3. Separate SUBDIRS provides easy points to make parts of the build conditional. If you’re adept with AM_CONDITIONALs this is no big deal, but it’s certainly easier to make a single directory conditional instead of all the targets it encompasses.

    4. Generally it’s a lot more maintenance than the standard recursive system for the previous points. If the reason to have the autotools is to reduce build system maintenance, then going non-recursive absolutely has the opposite effect. For a non-recursive build you almost certainly need someone around who’s quite familiar with the intricacies of the build.

    For an example of these drawbacks, I can simply point to the systemd non-recursive Makefile.

    http://cgit.freedesktop.org/systemd/systemd/tree/Makefile.am

    That thing is a freaking beast. I applaud them for pulling it off, but you’ll have a hard time convincing me that the maintenance burden is not high. I imagine a project of enlightenment’s size is similar. I guess that’s the tradeoff. For non-recursive make to provide a big enough payoff, you need to have a fairly big project. But a big project is where the build interactions become the most complex in a non-recursive build.

    1. Hi Dan,

      I agree that it comes with its own drawbacks. However I don’t think it’s worse than recursive automake. For small projects it’s very simple and once you are used to it, it’s just automatic. For big projects like enlightenment the pros pay off the extra work. And for sure there are experienced people to help on build-sys in big projects.

      Answering point by point:

      1. The namespace indeed can become very ugly and it’s extra verbosity. For the flags I have to disagree. It’s the same thing you should have done in the recursive automake. CFLAGS/LDFLAGS per target is the right thing to do in big projects. For the targets you have a common set of flags you put them in AM_CFLAGS. And this in inherited by all targets. If you happen to need to set the CFLAGS to a particular target the only thing you need to do is to define target_CFLAGS, prepending AM_CFLAGS. I don’t think this is any worse than defining an AM_CFLAGS per subdir, but rather it pushes you to do the right thing.

      2. Rather the opposite. By using a single automake the generated makefile has knowledge of all the dependencies. Why do you need to serialize the build in a per dir basis? Do you have an example?

      3. Yep, I agree. I tend to let things like doc and unit tests by using recursive makefiles. Remember it’s not “either one or the other”. We can mix both. For example, in enlightenment the only thing I converted was the build of modules. We still recurse in 22 directories. Some of them could be converted, but I don’t think there will be much benefit on this work so I simply keep as is.

      4. Well, it indeed depends on how experienced in autotools the people involved in the project are. Or how much they are willing to learn to have the benefits. IMO once the intricacies of non-recursive automake are learnt the biggest drawback will be only noise in the Makefile.am as you pointed out in systemd’s. From my experience it doesn’t add much more maintenance effort – it’s a lot of effort to convert, not to maintain.

      And, we are not talking about a small improvement in build time. For enlightenment it was 4 times smaller. You’ll have a hard time convincing me, as a developer, not to use it and having to wait that much more. And for packagers there are no cons, only pros.

      1. Hi Lucas,

        I probably overstated 2. I guess as long as the dependencies are all standard automake primitives then it’s no big difference (e.g. something in _LDADD is the same recursive or non-recursive). What I was thinking of was something where many targets depend on a built program or a built source file. Normally you could just seclude those in a directory and have it come first in SUBDIRS then everything would inherit from that. Otherwise, you’d have to state that built program/source/header as a prerequisite all over the place.

        I do agree that there can be very large build time gains on a big project with the non-recursive approach. For me, I’ve always felt that there’s a definite speed to complexity tradeoff. I always feel like as long as the build is fast enough that I don’t switch away to something else on incremental (dirty) runs then I’m happy. A project I’ve worked on a bunch that could definitely use the speed improvement is the xserver, but I also know there’s some nasty stuff that would make one large Makefile difficult to maintain. A project for someone…

  2. Note that you can achieve the same improvement in parallelism without necessarily eliminating the use of recursive make. As long as you make sure that any recursive calls to make use $(MAKE) rather than “make”, GNU make will arrange to propagate parallelism into the recursive calls.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>