Introduction to Remote Runners
A BuildBuddy remote runner is an execution environment that runs on a BuildBuddy executor. Our remote runners are optimized to run Bazel commands, and allow users to maintain a warm bazel instance in a secure execution environment managed by BuildBuddy. For example, this might look like a Firecracker microVM or an OCI container where you can run Bazel commands.
BuildBuddy remote runners have the following unique advantages:
- Colocation with BuildBuddy servers, ensuring a fast network connection between Bazel and BuildBuddy's cache & RBE servers.
- Running workloads in persistent execution environments using microVM snapshotting (on Linux) and persistent runners (on macOS). This allows reusing Bazel's in-memory analysis cache and local disk cache, achieving higher performance compared to remote caching alone.
There are two ways to use remote runners:
- BuildBuddy Workflows: Our continuous integration (CI) solution that runs Bazel builds and tests in response to git events (pull requests or pushes).
- Remote Bazel: a CLI tool that works exactly like the Bazel command, but runs Bazel on a remote workspace and streams the output back to the local machine.
See our blog post for more details on the motivation behind remote runners as well as some real-world results.
We currently support two products built on top of remote runners: Workflows and Remote Bazel.
Differences between Workflows and Remote Bazel
In many ways, Remote Bazel and Workflows are the same product and share much of the same backend code. Both are mechanisms to run code on remote runners. The primary difference is the entrypoint.
Workflows
Workflows are configured with a config YAML that is checked in to GitHub. Remote runs can be automatically triggered by GitHub events, like push and pull events. Workflows are commonly used as a Continuous Integration (CI) solution.
Workflows are a good fit if you:
- Have a static list of commands to run
- Want your commands checked in to your codebase for review
- Are exclusively using BuildBuddy to run CI and do not have another CI provider that can initiate commands
Remote Bazel
Remote Bazel can be configured by CURL request or by using the BuildBuddy CLI. Remote Bazel can be used as a Continuous Integration (CI) solution, or by developers in their daily workflows for use cases that are more dynamic than CI.
Remote Bazel is a good fit if you:
- Have a dynamic or frequently changing list of commands to run, that you do not want to check into your codebase
- You are continuing to use a legacy CI platform and want to integrate BuildBuddy into it
Benefits of remote runners
Colocation with BuildBuddy servers
Network latency is often the biggest bottleneck in many Bazel Remote Build Execution and Remote Caching setups. This is because Bazel's remote APIs require several chained RPCs due to dependencies between actions.
To address this bottleneck, BuildBuddy remote runners are executed in the same datacenters where BuildBuddy RBE and Cache nodes are deployed. This results in sub-millisecond round trip times to BuildBuddy's servers, minimizing the overhead incurred by Bazel's remote APIs.
Hosted, warm, Bazel instances
Running Bazel on most CI solutions is typically expensive and slow. There are several sources of overhead:
- When using Bazelisk, Bazel itself is re-downloaded and extracted on each CI run.
- The Bazel server starts from a cold JVM, meaning that it will be running unoptimized code until the JIT compiler kicks in.
- Bazel's analysis cache starts empty, which often means the entire workspace has to be re-scanned on each CI run.
- Any remote repositories referenced by the Bazel workspace all have to be re-fetched on each run.
- Bazel's on-disk cache starts completely empty, causing action re-execution or excess remote cache usage.
A common solution is to use something like actions/cache to store Bazel's cache for reuse between runs, but this solution is extremely data-intensive, as Bazel's cache can be several GB in size and consist of many individual files which are expensive to unpack from an archive. It also does not solve the problems associated with the Bazel server having starting from scratch.
By contrast, BuildBuddy uses a Bazel workspace reuse approach, similar to how Google's Build Dequeuing Service performs workspace selection:
A well-chosen workspace can increase the build speed by an order of magnitude by reusing the various cached results from the previous execution. [...] We have observed that builds that execute the same targets as a previous build are effectively no-ops using this technique
Bazel instance matching
To match remote runs to warm Bazel instances, BuildBuddy uses VM snapshotting powered by Firecracker on Linux, and a simpler runner-recycling based approach on macOS.