Thumbtack Engineering Engineering

Do More and `make` Less with GNU Make and Less.js

cat |

There has been much clamor of late about the proper way to configure front-end build systems.

So, when there are so many build systems out there, why make?

  • it is supported on almost all systems
  • it is easy to integrate with existing workflows
  • it is understood by many engineers from different domains
  • it has excellent dependency management

We have been using make in our frontend build process for over four years, and we love it.

Makefiles are not without their mysteries, however, and integrating them into a production build system is not entirely obvious.

This post is a walkthrough describing how we think about make for production CSS. We specifically address the issue of correctly, efficiently, and automatically managing the @import dependency chain.

Getting started with make

For small projects, you can express a build process with a short Makefile. Add a couple of lines to the Makefile that describe how to build the target CSS.

To build the target file app.css from app.less and any files it depends on, we can use a two-line Makefile:

build/app.css: src/app.less
    lessc src/app.less > build/app.css

(Note that Makefiles must use hard tabs for indentation.)

In this case, we indicate the target build/app.css, followed by a colon and the prerequisite src/app.less used to create that target. The line below is then the recipe invoked to actually run the build process.

Given the file above, running make will generate the file we're looking for. By default, make will output all of the commands it's running, so we can visually verify that the lessc compiler is invoked:

$ make
lessc src/app.less > build/app.css

(note you'll need to run gmake on BSD systems, as the Makefile described here is not BSD-compatible).

Now, if we were to type make again, nothing will happen. make uses the list of prerequisites to determine when it needs to rebuild a target. In this case, none of the prerequisites have changed, so there is no need to run the recipe. Let's try it out:

$ make
make: `build/app.css` is up to date.

But when we edit the source src/app.less and save it, we would expect that the target build/app.css would be rebuilt. Indeed, when we run make at this point, we see the target is re-built:

$ vim src/app.less # make edits and save
$ make
lessc src/app.less > build/app.css

This is the essence of a Makefile, and we're well on our way to having a reasonable build process.

Removing redundancy

As framed above, there is some redundancy in our Makefile. Let's take care of that.

For starters, we notice that the strings app.less and app.css are repeated in several places. Happily, make automatically defines two variables we can use: $@, which will contain the file we're building, and $<, the first dependency:

build/app.css: src/app.less
    lessc $< > $@

Let's consider a more complex example, where we have not only app.css but a separate CSS file for our application's landing page:

build/app.css: src/app.less
    lessc $< > $@

build/landing.css: src/landing.less
    lessc $< > $@

Again, we find ourselves with some obvious redundancy. Happily again, make lets us define patterns, instructing it how to build any CSS file from any LESS file:

build/%.css: src/%.less
    lessc $< > $@

Previously, make interpreted the single build target as its default, and invoking make would simply build that target. Now, however, when we run make in the shell, there is no default build target:

$ make
make: *** No targets.  Stop.

One way to specify a target to make is to just pass it on the command line:

$ make build/app.css build/landing.css
lessc src/app.less > build/app.css
lessc src/landing.less > build/landing.css

We won't want to list all the CSS files to build on the command line every time, so instead, we'll ask make to build a fake file called all, and have all depend on all the actual CSS files we need built:

build/%.css: src/%.less
    lessc $< > $@

all: build/app.css build/landing.css

Then, running make:

$ make
lessc src/app.less > build/app.css
lessc src/landing.less > build/landing.css

@import and the dependency chain

One of the great features of modern CSS is that we can use @import to build up CSS files from smaller files that contain variables, mixins, and rules. As a result, any @import statements imply a dependency.

/* app.less */
@import "imports/colors.less";
@import "imports/layout.less";

/* app.css specific rules... */

When any of the imported files changes, we know that app.css should be rebuilt. So, we need to inform make about this dependency chain. As always, make will only rebuild the files that need to be rebuilt.

build/%.css: src/%.less
    lessc $< > $@

build/app.css: src/imports/colors.less src/imports/layout.less

build/landing.css: src/imports/colors.less

all: build/app.css build/landing.css

(Note that the prerequisites for a target can be specified across multiple lines. make will, for example, understand that build/app.css depends on three files.)

As you can imagine from this simple case, managing dependencies across a large project quickly becomes is an exercise in tedium and is inevitably prone to error.

At Thumbtack, for example, we manage around 100 production CSS files with complex dependency chains. Building all CSS files from scratch requires upwards of 60 seconds, so it's important for developer productivity that our development build process only rebuild out of date files and no more.

Conveniently, since version 1.4, lessc supports a -M flag to output dependency lists that are dervied from the @import chain. Let's try it out:

$ lessc -M src/app.less build/app.css
build/app.css: src/app.less src/imports/colors.less src/imports/layout.less

(Note that you need to pass in an additional argument that will be used as the name of the target in the Makefile. In the example above we use the name of the built file build/app.css).

We can save the output of that command into another Makefile, which we will then include with a directive into our top-level Makefile. We'll make use of one more automatic variable $* to match the pattern from the target's %:

build/%.css: src/%.less
    lessc -M $< $@ > $*.d
    lessc $< > $@

-include app.d landing.d

all: build/app.css build/landing.css

The hyphen before include will inform make to ignore these files if they don't exist. This will likely be true the first time we run make.

We are using the .d extension to indicate dependency files.

This Makefile accomplishes our goal of automatic dependency management. make and lessc are updating our dependencies for us every time we run make.

To confirm, we can edit one of the imported CSS files and re-run make. We expect both of our CSS files to be rebuilt:

$ vim src/imports/colors.less # edit and save...
$ make
lessc src/app.less > build/app.css
lessc src/landing.less > build/landing.css

Excellent, we've made great progress. There's just one more thing. What happens when we delete a dependency altogether.

$ rm src/imports/colors.less
$ vim src/landing.less # remove reference to colors.less
$ vim src/app.less # remove reference to colors.less
$ make
make: *** No rule to make target `src/imports/colors.less', needed by `build/app.css'.  Stop.

The .d file still holds a reference to the now-missing file. make is correctly informing us that its understanding of the dependency graph indicates an error. However, we know that the .d file is simply out of date.

It turns out there is a very important paragraph in the GNU make manual (documented also by Scott McPeak):

If a rule has no prerequisites or recipe, and the target of the rule is a nonexistent file, then make imagines this target to have been updated whenever its rule is run. This implies that all targets depending on this one will always have their recipe run.

In other words, by following this advice, we can have make ignore missing prerequisites.

Our app.d dependency file, therefore, could be better expressed as follows:

build/app.css: src/app.less src/imports/colors.less src/imports/layout.less
src/app.less:
src/imports/colors.less:
src/imports/layout.less:

To make this happen automatically, we alter the recipe for the CSS files to append to the related .d files rules with no prerequisites or recipe. This is accomplished with the following line (commented for clarity):

# pipe the list of prerequisites
sed -e 's/^[^:]*: *//' < build/app.d | \
    # into `tr` and split them with newlines
    tr -s ' ' '\n' | \
    # put a colon at the end of each line
    sed -e 's/$$/:/' \
    # and concatenate back into the dependency file
    >> build/app.d

The resulting Makefile is therefore:

build/%.css: src/%.less
    lessc -M $< $@ > $*.d
    sed -e 's/^[^:]*: *//' < build/$*.d | \
        tr -s ' ' '\n' | \
        sed -e 's/$$/:/' \
        >> build/$*.d
    lessc $< > $@

-include app.d landing.d

all: build/app.css build/landing.css

And with that, we have automatic dependency management for our CSS files with make.

One more thing

We see now that we have manually listed the .d and .css files. We can use make's powerful shell and pattern substitution tools to handle this automatically for us:

SOURCES = $(shell ls src/*.less)
TARGETS = $(patsubst src/%.less,build/%.css,$(SOURCES))

-include $(TARGETS:.css=.d)

all: $(TARGETS)

Conclusion

make is a powerful tool for your front-end build environment. We've demonstrated that with make we can

  • Mimic the dependency graph from @imports automatically in Makefiles
  • Rebuild dependent targets when their dependencies change
  • Rebuild dependent targets when dependencies are added
  • Rebuild dependent targets when dependencies are deleted
  • Only build the files that are out of date (and no more)

Wrapping a Makefile with a service (like Watchman) is now trivial, can you can always be assured your CSS files are perfectly up to date.

Thanks to Justin Tulloss for writing about his recent success with make and inspiring this post.

References: