Prerequisites

What You'll Learn

We'll be working through a contrived example writing a build definition for wc from core utils. In doing so you'll:

What if I get stuck?

The final result of running through this codelab can be found here for reference. If you really get stuck you can find us on gitter!

Before we jump into writing custom build definitions, let me introduce you to genrule(), the generic build rule. Let's just create a new project and initialise Please in it:

$ mkdir custom_rules && cd custom_rules
$ plz init --no_prompt

Then create a BUILD file in the root of the repository like so:

BUILD

genrule(
    name = "word_count",
    srcs = ["file.txt"],
    deps = [],
    cmd = "wc $SRC > $OUT",
    outs = ["file.wc"],
)

Then create file.txt:

$ echo "the quick brown fox jumped over the lazy dog" > file.txt

and build it:

$ plz build //:word_count
Build finished; total time 70ms, incrementality 0.0%. Outputs:
//:word_count:
  plz-out/gen/file.wc

$ cat plz-out/gen/file.wc
 1  9 45 file.txt

So what's going on?

Here we've used one of the built-in rules, genrule(), to run a custom command. genrule() can take a number of parameters, most notably: the name of the rule, the inputs (sources and dependencies), its outputs, and the command we want to run. The full list of available arguments can be found on the genrule() documentation.

Here we've used it to count the number of words in file.txt. Please has helpfully set up some environment variables that help us find our inputs, as well as where to put our outputs:

For a complete list of available variables, see the build env docs.

The command wc $SRC > $OUT is therefore translated into wc file.txt > file.wc and we can see that the output of the rule has been saved to plz-out/gen/file.wc.

One of the key features of Please is that builds are hermetic, that is, commands are executed in an isolated and controlled environment. Rules can't access files or env vars that are not explicitly made available to them. As a result, incremental builds very rarely break when using Please.

Considering this, debugging builds would be quite hard if we couldn't play around in this build environment. Luckily, Please makes this trivial with the --shell flag:

$ plz build --shell :word_count
Temp directories prepared, total time 50ms:
  //:word_count: plz-out/tmp/word_count._build
    Command: wc $SRC > $OUT

bash-4.4$ pwd
<snip>/plz-out/tmp/word_count._build

bash-4.4$ wc $SRC > $OUT

bash-4.4$ cat $OUT
 1  9 45 file.txt

As we can see, Please has prepared a temporary directory for us under plz-out/tmp, and put us in a true-to-life bash environment. You may run printenv, to see the environment variables that Please has made available to us:

bash-4.4$ printenv
OS=linux
ARCH=amd64
LANG=en_GB.UTF-8
TMP_DIR=<snip>/plz-out/tmp/word_count._build
CMD=wc $SRC > $OUT
OUT=<snip>/plz-out/tmp/word_count._build/file.wc
TOOLS=
SRCS=file.txt
PKG=
CONFIG=opt
PYTHONHASHSEED=42
SRC=file.txt
OUTS=file.wc
PWD=<snip>/plz-out/tmp/word_count._build
HOME=<snip>/plz-out/tmp/word_count._build
NAME=word_count
TMPDIR=<snip>/plz-out/tmp/word_count._build
BUILD_CONFIG=opt
XOS=linux
XARCH=x86_64
SHLVL=1
PATH=<snip>/.please:/usr/local/bin:/usr/bin:/bin
GOOS=linux
PKG_DIR=.
GOARCH=amd64
_=/usr/bin/printenv

As you can see, the rule doesn't have access to any of the variables from the host machine. Even $PATH has been set based on configuration in .plzconfig:

The --shell flag works for all targets (except filegroups), which of course means any of the built-in rules! Note, --shell also works on plz test. You can plz build --shell //my:test to see how the test is built, and then plz test --shell //my:test to see how it will be run.

We've managed to write a custom rule to count the number of words in file.txt, however, we have no way of reusing this, so let's create a wordcount() build definition!

A build definition is just a function that creates one or more build targets which define how to build something. These are typically defined inside .build_def files within your repository. Let's just create a folder for our definition:

build_defs/word_count.build_defs

def word_count(name:str, file:str) -> str:
    return genrule(
        name = name,
        srcs = [file],
        outs = [f"{name}.wc"],
        cmd = "wc $SRC > $OUT",
    )

We then need some way to access these build definitions from other packages. To do this, we typically use a filegroup:

build_defs/BUILD

filegroup(
    name = "word_count",
    srcs = ["word_count.build_defs"],
    visibility = ["PUBLIC"],
)

We can then use this in place of our genrule():

BUILD

subinclude("//build_defs:word_count")

word_count(
    name = "word_count",
    file = "file.txt",
)

And check it still works:

$ plz build //:word_count
Build finished; total time 30ms, incrementality 100.0%. Outputs:
//:word_count:
  plz-out/gen/word_count.wc

subinclude()

Subinclude is primarily used for including build definitions into your BUILD file. It can be thought of like a Python import except it operates on a build target instead. Under the hood, subinclude parses the output of the target and makes the top-level declarations available in the current package's scope.

The build target is usually a filegroup, however, this doesn't have to be the case. In fact, the build target can be anything that produces parsable outputs.

It's almost always a bad idea to build anything as part of a subinclude. These rules will be built at parse time, which can be hard to debug, but more importantly, will block the parser while it waits for that rule to build. Use non-filegroup subincludes under very careful consideration!

Right now we're relying on wc to be available on the configured path. This is a pretty safe bet, however, Please provides a powerful mechanism for managing tools, so let's over-engineer this:

build_defs/word_count.build_defs

def word_count(name:str, file:str, wc_tool:str="wc") -> str:
    return genrule(
        name = name,
        srcs = [file],
        outs = [f"{name}.wc"],
        cmd = "$TOOLS_WC $SRC > $OUT",
        tools = {
            "WC": [wc_tool],
        }
    )

Here we've configured our build definition to take the word count tool in as a parameter. This is then passed to genrule() via the tools parameter. Please has set up the $TOOLS_WC environment variable which we can used to locate our tool. The name of this variable is based on the key in this dictionary.

In this contrived example, this may not seem very useful, however, Please will perform some important tasks for us:

Custom word count tool

Currently, our word count rule doesn't just get the word count: it also gets the character and line count as well. I mentioned that these can be build rules so let's create a true word count tool that counts just words:

tools/wc.sh

#!/bin/bash

wc -w $@

tools/BUILD

sh_binary(
    name = "wc",
    main = "wc.sh",
    visibility = ["PUBLIC"],
)

and let's test that out:

$ plz run //tools:wc -- file.txt
9 file.txt

Brilliant! We can now use this in our build rule like so:

BUILD

subinclude("//build_defs:word_count")

word_count(
    name = "lines_words_and_chars",
    file = "file.txt",
)

word_count(
    name = "just_words",
    file = "file.txt",
    wc_tool = "//tools:wc",
)

and check it all works:

$ plz build //:lines_words_and_chars //:just_words
Build finished; total time 30ms, incrementality 100.0%. Outputs:
//:lines_words_and_chars:
  plz-out/gen/lines_words_and_chars.wc
//:just_words:
  plz-out/gen/just_words.wc

$ cat plz-out/gen/lines_words_and_chars.wc
1  9 45 file.txt

$ cat plz-out/gen/just_words.wc
9 file.txt

Right now, we have to specify the new word count tool each time we use our build definition! Let's have a look at how we can configure this in our .plzconfig instead:

.plzconfig

[Buildconfig]
word-count-tool = //tools:wc

The [buildconfig] section can be used to add configuration specific to your project. By adding the word-count-tool config option here, we can use this in our build definition:

build_defs/word_count.build_defs

def word_count(name:str, file:str, wc_tool:str=CONFIG.WORD_COUNT_TOOL) -> str:
    return genrule(
        name = name,
        srcs = [file],
        outs = [f"{name}.wc"],
        cmd = "$TOOLS_WC $SRC > $OUT",
        tools = {
            "WC": [wc_tool],
        }
    )

CONFIG.setdefault('WORD_COUNT_TOOL', 'wc')

Here we've set the default value for wc_tool to CONFIG.WORD_COUNT_TOOL, which will contain our config value from .plzconfig. What if that's not set though? That's why we also set a sensible default configuration value with CONFIG.setdefault('WORD_COUNT_TOOL', 'wc')!

We then need to update our build rules:

BUILD

subinclude("//build_defs:word_count")

word_count(
    name = "lines_words_and_chars",
    file = "file.txt",
    wc_tool = "wc",
)

word_count(
    name = "just_words",
    file = "file.txt",
)

and check it all works:

$ plz build //:lines_words_and_chars //:just_words
Build finished; total time 30ms, incrementality 100.0%. Outputs:
//:lines_words_and_chars:
  plz-out/gen/lines_words_and_chars.wc
//:just_words:
  plz-out/gen/just_words.wc

$ cat plz-out/gen/lines_words_and_chars.wc
1  9 45 file.txt

$ cat plz-out/gen/just_words.wc
9 file.txt

Congratulations! You've written your first build definition! While contrived, this example demonstrates most of the mechanisms used to create a rich set of build definitions for a new language or technology. To get a better understanding of build rules, I recommend reading through the advanced topics on please.build.

If you create something you believe will be useful to the wider world, we might be able to find a home for it in the pleasings repo!

If you get stuck, jump on gitter and we'll do our best to help you!