Persistent workers

Please's usual model for build targets is that they're built in isolation from one another, by executing a command in a known directory that generates the expected outputs. In most cases this is great since it helps guarantee reproducibility, but sometimes it can be less than ideal - say your command has quite a lot of overhead to start up which you'd like to amortise between multiple invocations.

A motivating example here was the Java compiler which has a lot more startup overhead than natively implemented compilers (e.g. gcc, clang or go tool compile). Most Java build tools mitigate this by starting a single instance of it which is then reused to compile multiple packages.

Please supports a similar concept; it can be instructed to start workers as inferior processes that it manages the lifetime of. They are specified on a rule as follows:

        name = "my_rule",
        srcs = ["in.txt"],
        outs = ["out.txt"],
        tools = ["//tools:my_worker"],
        cmd = "$(worker //tools:my_worker) --do_work",
The worker itself must be marked as binary and will be built first if needed (it can also be a simple command if the tool is available globally).


Please communicates with the tool over stdin / stdout as a series of JSON messages. The structures in each direction are documented in src/worker/types.go; essentially it gets one message per build target with some extra metadata, and responds describing whther it was successful or not.
There's a minimal implementation of a worker in test/workers/worker.go, and a more complete example implementing a persistent Java compiler in the please-java repository.

Any messages that the process prints to its stderr will be consumed by Please and surfaced as errors; this provides a channel for the process to warn if things are going seriously wrong for some reason (i.e. more than just normal compile failures.


A similar mechanism exists for tests to share resources in the nature of common fixture setup, but that can be shared between multiple separate test targets. They work in very much the same way as when building and the same principles above apply.

The main conceptual difference is that the test worker is not responsible for actually executing the tests; Please still does that, and the worker simply does whatever setup is needed and responds back indicating when each test should begin.