Persistent workers
Please's usual model for build targets is that they're built in isolation from one another, by executing a command in a known directory that generates the expected outputs. In most cases this is great since it helps guarantee reproducibility, but sometimes it can be less than ideal - say your command has quite a lot of overhead to start up which you'd like to amortise between multiple invocations.
A motivating example here was the Java compiler which has a lot more startup
overhead than natively implemented compilers (e.g.
gcc
, clang
or
go tool compile
). Most Java build tools mitigate
this by starting a single instance of it which is then reused to compile
multiple packages.
Please supports a similar concept; it can be instructed to start workers as inferior processes that it manages the lifetime of. They are specified on a rule as follows:
genrule(
name = "my_rule",
srcs = ["in.txt"],
outs = ["out.txt"],
tools = ["//tools:my_worker"],
cmd = "$(worker //tools:my_worker) --do_work",
)
The worker itself must be marked as binary and will be built first if needed (it can also be a simple command if the tool is available globally).
Communication
Please communicates with the tool over stdin / stdout as a series of JSON
messages. The structures in each direction are documented in
src/worker/types.go; essentially it gets one message per build target with some extra
metadata, and responds describing whther it was successful or not.
There's a minimal implementation of a worker in
test/workers/worker.go, and a more complete example implementing a persistent Java compiler in
the
please-java
repository.
Any messages that the process prints to its stderr will be consumed by Please and surfaced as errors; this provides a channel for the process to warn if things are going seriously wrong for some reason (i.e. more than just normal compile failures.
Tests
A similar mechanism exists for tests to share resources in the nature of common fixture setup, but that can be shared between multiple separate test targets. They work in very much the same way as when building and the same principles above apply.
The main conceptual difference is that the test worker is not responsible for actually executing the tests; Please still does that, and the worker simply does whatever setup is needed and responds back indicating when each test should begin.