What is Maelstrom?

Maelstrom is a Rust test runner, built on top of a general-purpose clustered job runner. Maelstrom packages your Rust tests into hermetic micro-containers, then distributes them to be run on an arbitrarily large cluster of test-runners, or locally on your machine. You might use Maelstrom to run your tests because:

• It's easy. Maelstrom functions as a drop-in replacement for cargo test, so in most cases, it just works.
• It's reliable. Maelstrom runs every test hermetically in its own lightweight container, eliminating confusing errors caused by inter-test or implicit test-environment dependencies.
• It's scalable. Maelstrom can be run as a cluster. You can add more worker machines to linearly increase test throughput.
• It's fast. In most cases, Maelstrom is faster than cargo test, even without using clustering.
• It's clean. Maelstrom has a from-scratch, rootless container implementation (not relying on Docker or RunC), optimized to be low-overhead and start quickly.
• It's Rusty. The whole project is written in Rust.

We started with a Rust test runner, but Maelstrom's underlying job execution system is general-purpose. We will add support for other languages' test frameworks in the near future. We have also provided tools for adventurous users to run arbitrary jobs, either using a command-line tool or a gRPC-based SDK.

The project is currently Linux-only (x86 and ARM), as it relies on namespaces to implement containers.

Structure of This Book

This book will start out covering how to install Maelstrom. Next, it will cover common concepts that are applicable to all Maelstrom components. After that, there are in-depth sections for each of the four binaries: cargo-maelstrom, maelstrom-broker, maelstrom-worker, and maelstrom-run.

There is no documentation yet for the gRPC API. Contact us if you're interested in using it, and we'll help get you started.

Installation

Maelstrom consists of the following binaries:

• cargo-maelstrom: A Rust test runner. This can be run in standalone mode — where tests will be executed on the local machine — or in clustered mode. In standalone mode, no other Maelstrom binaries need be installed, but in clustered mode, there must be a broker and some workers available on the network.
• maelstrom-broker: The Maelstrom cluster scheduler. There must be one of these per Maelstrom cluster.
• maelstrom-worker: The Maelstrom cluster worker. This must be installed on the machines in the cluster that will actually run jobs (i.e. tests).
• maelstrom-run: A Maelstrom client for running arbitrary commands on a Maelstrom cluster. While this binary can run in standalone mode, it's only really useful in clustered mode.

The installation process is virtually identical for all binaries. We'll demonstrate how to install all the binaries in the following sections. You can pick and choose which ones you actually want to install.

Maelstrom currently only supports Linux.

Installing From Pre-Built Binaries

The easiest way to install Maelstrom binaries is to use cargo-binstall:

cargo binstall cargo-maelstrom
cargo binstall maelstrom-worker
cargo binstall maelstrom-broker
cargo binstall maelstrom-run

These commands retrieve the pre-built binaries from the Maelstrom GitHub release page. If you don't have cargo-binstall, you can just manually install the binaries from the releases page. For example:

wget -q -O - https://github.com/maelstrom-software/maelstrom/releases/latest/download/cargo-maelstrom-x86_64-unknown-linux-gnu.tgz | tar xzf -

This will download and extract the latest release of cargo-maelstrom for x86 Linux.

Installing Using Nix

We have a nix.flake file, so you can install all Maelstrom binaries with something like:

nix profile install github:maelstrom-software/maelstrom

Our Nix flake doesn't currently have the ability to install individual binaries.

Installing From Source With cargo install

Maelstrom binaries can be built from source using cargo install:

cargo install cargo-maelstrom
cargo install maelstrom-worker
cargo install maelstrom-run

However, maelstrom-broker requires some extra dependencies be installed before it can be built from source:

rustup target add wasm32-unknown-unknown
cargo install wasm-opt
cargo install maelstrom-broker

Common Concepts

This chaper covers concepts that are common to Maelstrom as a whole. Later chapers will cover specific programs.

Configuration Values

All Maelstrom executables are configured through "configuration values". Configuration values can be set through commmand-line options, environment variables, or configuration files.

Each configuration value has a type, which is either string, number, or boolean.

Imagine a configuration value named config-value in a program called maelstrom-prog. This configuration value can be specified via:

• The --config-value command-line option.
• The MAELSTROM_PROG_CONFIG_VALUE environment variable.
• The config-value key in a configuration file.

Command-Line Options

Configuration values set on the command line override settings from environment variables or configuration files.

There is currently no way to set a boolean configuration value to false from the command-line.

Environment Variables

Configuration values set via environment variables override settings from configuration files, but are overridden by command-line options.

The environment variable name is created by converting the configuration value to "screaming snake case", and prepending a program-specific prefix. Image that we're evaluating configuration values for a program name "maelstrom-prog":

Note that you don't need to put quotation marks around string values. You also can set boolean values to either true or false.

Configuration Files

Configuration files are in TOML format. In TOML files, configuration values map to keys of the same name. Values types map to the corresponding TOML types. For example:

frob-name = "string"
frob-size = 42
enable-frobs = true
enable-qux = false

Maelstrom programs support the existence of multiple configuration files. In this case, the program will read each one in preference order, with the settings from the higher-preference files overriding those from lower-preference files.

Configuration File Search

By default, Maelstrom programs will use the XDG Base Directory Specification for searching for configuration files.

Specifically, any configuration file found in XDG_CONFIG_HOME has the higest preference, followed by those found in XDG_CONFIG_DIRS. If XDG_CONFIG_HOME is not set, or is empty, then ~/.config/ is used. Similiarly, if XDG_CONFIG_DIRS is not set, or is empty, then /etc/xdg/ is used.

Each program has a program-specific suffix that it appends to the directory it gets from XDG. This has the form maelstrom/prog, where prog is program-specific.

Finally, the program looks for a file named config.toml in these directories.

More concretely, these are where Maelstrom programs will look for configuration files:

For example, if neither XDG_CONFIG_HOME nor XDG_CONFIG_DIRS is set, then cargo-maelstrom will look for two configuration files:

• ~/.config/maelstrom/cargo-maelstrom/config.toml
• /etc/xdg/maelstrom/cargo-maelstrom/config.toml

Overriding Configuration File Location

Maelstrom programs also support the --config-file (-c) command-line option. If this option is provided, the specified configuration file, and only that file, will be used.

If --config-file is given - as an argument, then no configuration file is used.

Log Levels

Every Maelstrom program supports the log-level configuration value. The program will output log messages of the given severity or higher. This string configuration value must be one of the following:

The default value is "info".

Standard Command-Line Options

All Maelstrom programs support a standard base set of command-line options.

--help

The --help (or -h) command-line option will print out the program's command-line options, configuration values (including their associated environment variables), and configuration-file search path, then exit.

--version

The --version (or -v) command-line option will cause the program to print its software version, then exit.

--print-config

The --print-config (or -P) command-line option will print out all of the program's configuration values, then exit. This can be useful for validating configuration.

--config-file

The --config-file (or -c) command-line option is used to specify a specific configuration file, or specify that no configuration file should be used. See here for more details.

The Project Directory

Maelstrom clients have a concept of the "project directory". For cargo-maelstrom, it is the Cargo workspace root directory. For maelstrom-run, it is the current working directory.

The project directory is used to resolve local relative paths. It's also where the client will put the container tags lock file.

The Job Specification

Each job run by Maelstrom is defined by a Job Specification, or "job spec" for short. Understanding job specifications is important for understanding what's going on with your tests, for toubleshooting failing tests, and for understanding cargo-maelstrom's configuration directives and maelstrom-run's input format.

This chapter shows the specification and its related types in the Protocol Buffer format. This is done because it's a convenient format to use for documentation. You won't have to interact with Maelstrom at this level.

This is the Protocol Buffer for the JobSpec:

message JobSpec {
    string program = 1;
    repeated string arguments = 2;
    repeated string environment = 3;
    repeated LayerSpec layers = 4;
    repeated JobDevice devices = 5;
    repeated JobMount mounts = 6;
    bool enable_loopback = 7;
    bool enable_writable_file_system = 8;
    string working_directory = 9;
    uint32 user = 10;
    uint32 group = 11;
    optional uint32 timeout = 12;
}

program

This is the path of the program to run, relative to the working_directory. The job will complete when this program terminates, regardless of any other processes that have been started.

The path must be specified explicitly: the PATH environment variable will not be searched, even if it is provided.

The program is run as PID 1 in its own PID namespace, which means that it acts as the init process for the container. This shouldn't matter for most use cases, but if the program starts a lot of subprocesses, it may need to explicitly clean up after them.

The program is run as both session and process group leader.

arguments

These are the arguments to pass to program, excluding the name of the program itself. For example, to run cat foo bar, you would set program to "cat" and arguments to ["foo", "bar"].

environment

These are the environment variables passed to program. These are in the format expected by the execve syscall, which is "VAR=VALUE".

The PATH environment variable is not used when searching for program. You must provide the actual path to program.

layers

enum ArtifactType {
    Tar = 0;
    Manifest = 1;
}

message LayerSpec {
    bytes digest = 1;
    ArtifactType type = 2;
}

The file system layers specify what file system the program will be run with. They are stacked on top of each other, starting with the first layer, with later layers overriding earlier layers.

Each layer will be a tar file, or Maelstrom's equivalent called a "manifest file". These LayerSpec objects are usually generated with an AddLayerRequest, which is described in the next chapter. cargo-maelstrom and maelstrom-run provide ways to conveniently specify these, as described in their respective chapters.

devices

enum JobDevice {
    Full = 0;
    Fuse = 1;
    Null = 2;
    Random = 3;
    Tty = 4;
    Urandom = 5;
    Zero = 6;
}

These are the device files from /dev to add to the job's environment. Any subset can be specified.

Any specified device will be mounted in /dev based on its name. For example, Null would be mounted at /dev/null. For this to work, there must be a file located at the expected location in the container file system. In other words, if your job is going to specify Null, it also needs to have an empty file at /dev/null for the system to mount the device onto. This is one of the use cases for the "stubs" layer type.

mounts

enum JobMountFsType {
    Proc = 0;
    Tmp = 1;
    Sys = 2;
}

message JobMount {
    JobMountFsType fs_type = 1;
    string mount_point = 2;
}

These are extra file systems mounts put into the job's environment. They are applied in order, and the mount_point must already exist in the file system. Providing the mount point is one of the use cases for the "stubs" layer type.

The mount_point is relative to the root of the file system, even if there is a working_directory specified.

For more information about these file system types, see:

• Proc
• Tmp
• Sys

enable_loopback

Jobs are run completely disconnected from the network. Without this flag set, they don't even have a loopback device enabled, and thus cannot communicate to localhost/127.0.0.1/::1.

Enabling this flag allows communication on the loopack device.

enable_writable_file_system

By default, the whole file system the job sees will be read-only, except for any extra file systems specified in mounts.

Enabling this flag will make the file system writable. Any changes the job makes to the file system will be isolated, and will be thrown away when the job completes.

working_directory

This specifies the directory that program is run in.

user

This specifies the UID the program is run as.

Maelstrom runs all of its jobs in rootless containers, meaning that they don't require any elevated permissions on the host machines. All containers will be run on the host machine as the user running cargo-maelstrom, maelstrom-run, or maelstrom-worker, regardless of what this field is set as.

However, if this is field is set to 0, the program will have some elevated permissions within the container, which may be undesirable for some jobs.

group

The specifies the GID the program is run as. See user for more information.

Jobs don't have any supplemental GIDs, nor is there any way to provide them.

timeout

This specifies an optional timeout for the job, in seconds. If the job takes longer than the timeout, Maelstrom will terminate it and return the partial results. A value of 0 indicates an infinite timeout.

Layers

At the lowest level, a layer is just a tar file or a manifest. A manifest is a Maelstrom-specific file format that allows for file data to be transferred separately from file metadata. But for our purposes here, they're essentially the same.

As a user, having to specify every layer as a tar file would be very painful. For this reason, Maelstrom provides some conveniences for creating layers based on specification. Under the covers, there is an API that the Maelstrom clients cargo-maelstrom and maelstrom-run use for creating layers. This is what that API looks like:

message AddLayerRequest {
    oneof Layer {
        TarLayer tar = 1;
        GlobLayer glob = 2;
        PathsLayer paths = 3;
        StubsLayer stubs = 4;
        SymlinksLayer symlinks = 5;
    }
}

We will cover each layer type below.

tar

message TarLayer {
    string path = 1;
}

The tar layer type is very simple: The provided tar file will be used as a layer. The path is specified relative to the client.

prefix_options

message PrefixOptions {
    optional string strip_prefix = 1;
    optional string prepend_prefix = 2;
    bool canonicalize = 3;
    bool follow_symlinks = 4;
}

The paths and glob layer types support some options that can be used to control how the resulting layer is created. They apply to all paths included in the layer. These options can be combined, and in such a scenario you can think of them taking effect in the given order:

• follow_symlinks: Don't include symlinks, instead use what they point to.
• canonicalize: Use absolute form of path, with components normalized and symlinks resolved.
• strip_prefix: Remove the given prefix from paths.
• prepend_prefix Add the given prefix to paths.

Here are some examples.

follow_symlinks

If test/d/symlink is a symlink which points to the file test/d/target, and is specified with follow_symlinks, then Maelstrom will put a regular file in the container at /test/d/symlink with the contents of test/d/target.

canonicalize

If the client is executing in the directory /home/bob/project, and the layers/c/*.bin glob is specified with canonicalize, then Maelstrom will put files in the container at /home/bob/project/layers/c/*.bin.

Additionally, if /home/bob/project/layers/py is a symlink pointing to /var/py, and the layers/py/*.py glob is specified with canonicalize, then Maelstrom will put files in the container at /var/py/*.py.

strip_prefix

If layers/a/a.bin is specified with strip_prefix = "layers/", then Maelstrom will put the file in the container at /a/a.bin.

prepend_prefix

If layers/a/a.bin is specified with prepend_prefix = "test/", then Maelstrom will put the file in the container at /test/layers/a/a.bin.

glob

message GlobLayer {
    string glob = 1;
    PrefixOptions prefix_options = 2;
}

The glob layer type will include the files specified by the glob pattern in the layer. The glob pattern is executed by the client relative to the project directory. The glob pattern must use relative paths. The globset crate is used for glob pattern matching.

The prefix_options are applied to every matching path, as described above.

paths

message PathsLayer {
    repeated string paths = 1;
    PrefixOptions prefix_options = 2;
}

The paths layer type will include each file referenced by the specified paths. This is executed by the client relative to the project directory. Relative and absolute paths may be used.

The prefix_options are applied to every matching path, as described above.

If a path points to a file, the file is included in the layer. If the path points to a symlink, either the symlink or the pointed-to-file gets included, depending on prefix_options.follow_symlinks. If the path points to a directory, an empty directory is included.

To include a directory and all of its contents, use the glob layer type.

stubs

message StubsLayer {
    repeated string stubs = 1;
}

The stubs layer type is used to create empty files and directories, usually so that they can be mount points for devices or mounts.

If a string contains the { character, the bracoxide crate is used to perform brace expansion, transforming the single string into multiple strings.

If a string ends in /, an empty directory will be added to the layer. Otherwise, and empty file will be added to the layer. Any parent directories will also be created as necessary.

For example, the set of stubs ["/dev/{null,zero}", "/{proc,tmp}/", "/usr/bin/"] would result in a layer with the following files and directories:

• /dev/
• /dev/null
• /dev/zero
• /proc/
• /tmp/
• /usr/
• /usr/bin/

symlinks

message SymlinkSpec {
    string link = 1;
    string target = 2;
}

message SymlinksLayer {
    repeated SymlinkSpec symlinks = 1;
}

The symlinks layer is used to create symlinks. The specified links will be created, with the specified targets. Any parent directories will also be created, as necessary.

Local Worker

The cargo-maelstrom and maelstrom-run clients can run in "standalone mode". In this mode, they don't submit jobs to a Maelstrom cluster, but instead run the jobs locally, using a local worker.

Standalone mode is specified when no broker configuration value is provided.

Currently, clients won't use the local worker if they are connected to a cluster. We will change this in the future so that the local worker is utilized even when the client is connected to a cluster.

Clients have the following configuration values to configure their local workers:

cache-size

The cache-size configuration value specifies a target size for the cache. Its default value is 1 GB. When the cache consumes more than this amount of space, the worker will remove unused cache entries until the size is below this value.

It's important to note that this isn't a hard limit, and the worker will go above this amount in two cases. First, the worker always needs all of the currently-executing jobs' layers in cache. Second, the worker currently first downloads an artifact in its entirety, then adds it to the cache, then removes old values if the cache has grown too large. In this scenario, the combined size of the downloading artiface and the cache may exceed cache-size.

For these reasons, it's important to leave some wiggle room in the cache-size setting.

inline-limit

The inline-limit configuration value specifies how many bytes of stdout or stderr will be captured from jobs. Its default value is 1 MB. If stdout or stderr grows larger, the client will be given inline-limit bytes and told that the rest of the data was truncated.

In the future we will add support for the worker storing all of stdout and stderr if they exceed inline-limit. The client would then be able to download it "out of band".

slots

The slots configuration value specifies how many jobs the worker will run concurrently. Its default value is the number of CPU cores on the machine. In the future, we will add support for jobs consuming more than one slot.

Job States

Jobs transition through a number of states in their journey. This chapter explains those states.

Waiting for Artifacts

If a broker doesn't have all the required artifacts for a job when it is submitted, the job enters this state. The broker will notify the client of the missing artifacts, and wait for the client to transfer them. Once all artifacts have been received from the client, the job will proceed to the next state.

We think of all jobs initially entering this state, and then immediately transitioning to Pending if the broker has all of the artifacts. Also, local jobs immediately transition out of this state, since the worker is co-located with the client and has immediate access to all of the artifacts.

Pending

In the Pending state, the broker has the job and all of its artifacts, but hasn't yet found a free worker to execute the job. Jobs in this state are stored in a queue. Once a job reaches the front of the queue, and a worker becomes free, the job will be sent to the worker for execution.

Local jobs aren't technically sent to the broker. However, they still do enter a queue waiting to be submitted to the local worker, which is pretty similar to the situation for remote jobs. For that reason, we lump local and remote jobs together in this state.

Running

A Running job has been set to the worker for execution. The worker could be executing the job, or it could be transferring some artifacts from the broker. In the future, we will likely split this state apart into the various different sub-states. If a worker disconnects from the broker, the broker moves all jobs that were assigned to that worker back to the [Pending] state.

Completed

Jobs in this state have been executed to completion by a worker.

Container Images

Maelstrom clients support building job specifications based off of standard OCI container images stored on Docker Hub. This can be done with cargo-maelstrom's image directive field, or cargo-run's image field.

When an image is specified this way, the client will first download it locally into its cache directory. It will then use the internal bits of the OCI image — most importantly the file-system layers — to create the job specification for the job.

Images can be specified with tags. If no tag is provided, the latest tag is used.

For the purposes of reproducable jobs, clients will resolve and "lock" a tag, so that jobs always specify an exact image that doesn't change over time. See this section for more details.

Cached Container Images

Container images are cached on the local file system. Maelstrom uses the $XDG_CACHE_HOME/maelstrom/containers directory if XDG_CACHE_HOME is set and non-empty. Otherwise, it uses ~/.cache/maelstrom/containers. See the XDG Base Directories specification for more information.

Lock File

When a client first resolves a container registry tag, it stores the result in a local lock file. Subsequently, it will use the exact image specified in the lock file instead of resolving the tag again. This guarantees that subsequent runs use the same images as previous runs.

The local lock file is maelstrom-container-tags.lock, stored in the project directory. It is recommended that this file be committed to revision control, so that others in the project, and CI, use the same images when running tests.

To update a tag to the latest version, remove the corresponding line from the lock file and then run the client.

cargo-maelstrom

cargo-maelstrom is a replacement for cargo test which runs tests in lightweight containers, either locally or on a distributed cluster. Since each test runs in its own container, it is isolated from the computer it is running on and from other tests.

cargo-maelstrom is designed to be run as a custom Cargo subcommand. One can either run it as cargo-maelstrom or as cargo maelstrom.

For a lot of projects, cargo-maelstrom will run all tests successfully, right out of the box. Some tests, though, have external dependencies that cause them to fail when run in cargo-maelstrom's default, stripped-down containers. When this happens, it's usually pretty easy to configure cargo-maelstrom so that it invokes the test in a container that contains all of the necessary dependencies. The Job Specification chaper goes into detail about how to do so.

Test Filter Patterns

There are times when a user needs to concisely specify a set of tests to cargo-maelstrom. One of those is on the command line: cargo-maelstrom can be told to only run a certain set of tests, or to exclude some tests. Another is the filter field of maelstrom-test.toml directives. This is used to choose which tests a directive applies too.

In order to allow users to easily specify a set of tests to cargo-maelstrom, we created the domain-specific pattern language described here.

If you are a fan of formal explanations check out the BNF. Otherwise, this page will attempt to give a more informal explanation of the language.

Simple Selectors

The most basic patterns are "simple selectors". These are only sometimes useful on their own, but they become more powerful when combined with other patterns. Simple selectors consist solely of one of the these identifiers:

Simple selectors can optionally be followed by (). That is, library() and library are equivalent patterns.

Compound Selectors

"Compound selector patterns" are patterns like package.equals(foo). They combine "compound selectors" with "matchers" and "arguments". In our example, package is the compound selector, equals is the matcher, and foo is the argument.

These are the possible compound selectors:

Documentation on the various types of targets in cargo can be found here.

These are the possible matchers:

Compound selectors and matchers are separated by . characters. Arguments are contained within delimeters, which must be a matched pair:

The compound selectors binary, benchmark, example, and test will only match if the test is from a target of the specified type and the target's name matches. In other words, binary.equals(foo) can be thought of as shorthand for the compound pattern (binary && binary.equals(foo)).

Let's put this all together with some examples:

Compound Expressions

Selectors can be joined together with operators to create compound expressions. These operators are:

The "logical difference" action is defined as follows: A - B == A && !B.

As an example, to select tests named foo or bar in package baz:

(name.equals(foo) || name.equals(bar)) && package.equals(baz)

As another example, to select tests named bar in package baz or tests named foo from any package:

name.equals(foo) || (name.equals(bar) && package.equals(baz))

Abbreviations

Selector and matcher names can be shortened to any unambiguous prefix.

For example, the following are all the same

name.equals(foo)
name.eq(foo)
n.eq(foo)

We can abbreviate name to n since no other selector starts with "n", but we can't abbreviate equals to e because there is another selector, ends_with, that also starts with an "e".

Test Pattern DSL BNF

Included on this page is the Backus-Naur form notation for the DSL

pattern                := or-expression
or-expression          := and-expression
                       |  or-expression or-operator and-expression
or-operator            := "|" | "||" | "or"
and-expression         := not-expression
                       |  and-expression and-operator not-expression
                       |  and-expression diff-operator not-expression
and-operator           := "&" | "&&" | "and" | "+"
diff-operator          := "\" | "-" | "minus"
not-expression         := simple-expression
                       |  not-operator not-expression
not-operator           := "!" | "~" | "not"
simple-expression      := "(" or-expression ")"
                       |  simple-selector
                       |  compound-selector
simple-selector        := simple-selector-name
                       |  simple-selector-name "(" ")"
simple-selector-name   := "all" | "any" | "true"
                       |  "none" | "false"
                       |  "library"
                       |  compound-selector-name
compound-selector      := compound-selector-name "." matcher-name matcher-parameter
compound-selector-name := "name" | "binary" | "benchmark" | "example" |
                          "test" | "package"
matcher-name           := "equals" | "contains" | "starts_with" | "ends_with" |
                          "matches" | "globs"
matcher-parameter      := <punctuation mark followed by characters followed by
                           matching punctuation mark>

Job Specification: maelstrom-test.toml

The file maelstrom-test.toml in the workspace root is used to specify to cargo-maelstrom what job specifications are used for which tests.

This chapter describes the format of that file and how it is used to set the job spec fields described here.

Default Configuration

If there is no maelstrom-test.toml in the workspace root, then cargo-maelstrom will run with the following defaults:

# Because it has no `filter` field, this directive applies to all tests.
[[directives]]

# Copy any shared libraries the test depends on along with the binary.
include_shared_libraries = true

# This layer just includes files and directories for mounting the following
# file-systems and devices.
layers = [
    { stubs = [ "/{proc,sys,tmp}/", "/dev/{full,null,random,urandom,zero}" ] },
]

# Provide /tmp, /proc, /sys. These are used pretty commonly by tests.
mounts = [
    { fs_type = "tmp", mount_point = "/tmp" },
    { fs_type = "proc", mount_point = "/proc" },
    { fs_type = "sys", mount_point = "/sys" },
]

# Mount these devices in /dev/. These are used pretty commonly by tests.
devices = ["full", "null", "random", "urandom", "zero"]

Initializing maelstrom-test.toml

It's likely that at some point you'll need to adjust the job specs for some tests. At that point, you're going to need an actual maelstrom-test.toml. Instead of starting from scratch, you can have cargo-maelstrom create one for you:

cargo maelstrom --init

This will create a maelstrom-test.toml file, unless one already exists, then exit. The resulting maelstrom-test.toml will match the default configuration. It will also include some commented-out examples that may be useful.

Directives

The maelstrom-test.toml file consists of a list of "directives" which are applied in order. Each directive has some optional fields, one of which may be filter. To compute the job spec for a test, cargo-maelstrom starts with a default spec, then iterates over all the directives in order. If a directive's filter matches the test, the directive is applied to the test's job spec. Directives without a filter apply to all tests. When it reaches the end of the configuration, it pushes one or two more layers containing the test executable, and optionally all shared library dependencies (see here for details). The job spec is then used for the test.

There is no way to short-circuit the application of directives. Instead, filters can be used to limit scope of a given directive.

To specify a list of directives in TOML, we use the [[directives]] syntax. Each [[directives]] line starts a new directive. For example, this snippet specifies two directives:

[[directives]]
include_shared_libraries = true

[[directives]]
filter = "package.equals(maelstrom-util) && name.equals(io::splicer)"
added_mounts = [{ fs_type = "proc", mount_point = "/proc" }]
added_layers = [{ stubs = [ "proc/" ] }]

The first directive applies to all tests, since it has no filter. It sets the include_shared_libraries psuedo-field in the job spec. The second directive only applies to a single test named io::splicer in the maelstrom-util package. It adds a layer and a mount to that test's job spec.

Directive Fields

This chapter specifies all of the possible fields for a directive. Most, but not all, of these fields have an obvious mapping to job-spec fields.

filter

This field must be a string, which is interpretted as a test filter pattern. The directive only applies to tests that match the filter. If there is no filter field, the directive applies to all tests.

Sometimes it is useful to use multi-line strings for long patterns:

[[directives]]
filter = """
package.equals(maelstrom-client) ||
package.equals(maelstrom-client-process) ||
package.equals(maelstrom-container) ||
package.equals(maelstrom-fuse) ||
package.equals(maelstrom-util)"""
layers = [{ stubs = ["/tmp/"] }]
mounts = [{ fs_type = "tmp", mount_point = "/tmp" }]

include_shared_libraries

[[directives]]
include_shared_libraries = true

This boolean field sets the include_shared_libraries job spec psuedo-field. We call it a psuedo-field because it's not a real field in the job spec, but instead determines how cargo-maelstrom will do its post-processing after computing the job spec from directives.

In post-processing, if the include_shared_libraries psuedo-field is false, cargo-maelstrom will only push a single layer onto the job spec. This layer will contain the test executable, placed in the root directory.

On the other hand, if the psuedo-field is true, then cargo-maelstrom will push two layers onto the job spec. The first will be a layer containing all of the shared-library dependencies for the test executable. The second will contain the test executable, placed in the root directory. (Two layers are used so that the shared-library layer can be cached and used by other tests.)

If the psuedo-field is never set one way or the other, then cargo-maelstrom will choose a value based on the layers field of the job spec. In this case, include_shared_libraries will be true if and only if layers is empty.

You usually want this pseudo-field to be true, unless you're using a container image for your tests. In that case, you probably want to use the shared libraries included with the container image, not those from the system running the tests.

image

Sometimes it makes sense to build your test's container from an OCI container image. For example, when we do integration tests of cargo-maelstrom, we want to run in an environment with cargo installed.

This is what the image field is for.

[[directives]]
filter = "package.equals(cargo-maelstrom)"
image.name = "rust"
image.use = ["layers", "environment"]

[[directives]]
filter = "package.equals(maelstrom-client) && test.equals(integration_test)"
image = { name = "alpine", use = ["layers", "environment"] }

The image field must be a table with two subfields: name and use.

The name sub-field specifies the name of the image. It must be a string. This will be used to find the image on Docker Hub.

The use sub-field must be a list of strings specifying what parts of the container image to use for the job spec. It must contain a non-empty subset of:

• "layers": The layers field of the job spec is replaced by the layers specified in the container image. These layers will be tar layers pointing to the tar files in the container depot.
• "environment": The environment field of the job spec is replaced by the environment specified in the container image.
• "working_directory": The working_directory field of the job spec is replaced by the working directory specified in the container image.

In the example above, we specified a TOML table in two different, equivalent ways for illustrative purposes.

layers

[[directives]]
layers = [
    { tar = "layers/foo.tar" },
    { paths = ["layers/a/b.bin", "layers/a/c.bin"], strip_prefix = "layers/a/" },
    { glob = "layers/b/**", strip_prefix = "layers/b/" },
    { stubs = ["/dev/{null, full}", "/proc/"] },
    { symlinks = [{ link = "/dev/stdout", target = "/proc/self/fd/1" }] }
]

This field provides an ordered list of layers for the job spec's layers field.

Each element of the list must be a table with one of the following keys:

• tar: The value must be a string, indicating the local path of the tar file. This is used to create a tar layer.
• paths: The value must be a list of strings, indicating the local paths of the files and directories to include to create a paths layer. It may also include fields from prefix_options (see below).
• glob: The value must be a string, indicating the glob pattern to use to create a glob layer. It may also include fields from prefix_options (see below).
• stubs: The value must be a list of strings. These strings are optionally brace-expanded and used to create a stubs layer.
• symlinks: The value must be a list of tables of link/target pairs. These strings are used to create a symlinks layer.

If the layer is a paths or glob layer, then the table can have any of the following extra fields used to provide the prefix_options:

• follow_symlinks: A boolean value. Used to specify follow_symlinks.
• canonicalize: A boolean value. Used to specify canonicalize.
• strip_prefix: A string value. Used to specify strip_prefix.
• prepend_prefix: A string value. Used to specify prepend_prefix.

For example:

[[directives]]
layers = [
    { paths = ["layers"], strip_prefix = "layers/", prepend_prefix = "/usr/share/" },
]

This would create a layer containing all of the files and directories (recursively) in the local layers subdirectory, mapping local file layers/example to /usr/share/example in the test's container.

This field can't be set in the same directive as image if the image.use contains "layers".

added_layers

This field is like layers, except it appends to the job spec's layers field instead of replacing it.

This field can be used in the same directive as an image.use that contains "layers". For example:

[[directives]]
image.name = "cool-image"
image.use = ["layers"]
added_layers = [
    { paths = [ "extra-layers" ], strip_prefix = "extra-layers/" },
]

This directive sets uses the layers from "cool-image", but with the contents of local extra-layers directory added in as well.

environment

[[directives]]
environment = {
    USER = "bob",
    RUST_BACKTRACE = "$env{RUST_BACKTRACE:-0}",
}

This field sets the environment field of the job spec. It must be a table with string values. It supports two forms of $ expansion within those string values:

• $env{FOO} evaluates to the value of cargo-maelstrom's FOO environment variable.
• $prev{FOO} evaluates to the previous value of FOO for the job spec.

It is an error if the referenced variable doesn't exist. However, you can use :- to provide a default value:

FOO = "$env{FOO:-bar}"

This will set FOO to whatever cargo-maelstrom's FOO environment variable is, or to "bar" if cargo-maelstrom doesn't have a FOO environment variable.

This field can't be set in the same directive as image if the image.use contains "environment".

added_environment

This field is like environment, except it updates the job spec's environment field instead of replacing it.

When this is provided in the same directive as the environment field, the added_environment gets evaluated after the environment field. For example:

[[directives]]
environment = { VAR = "foo" }

[[directives]]
environment = { VAR = "bar" }
added_environment = { VAR = "$prev{VAR}" }

In this case, VAR will be "bar", not "foo".

This field can be used in the same directive as an image.use that contains "environment". For example:

[[directives]]
image = { name = "my-image", use = [ "layers", "environment" ] }
added_environment = { PATH = "/scripts:$prev{PATH}" }

This prepends "/scripts" to the PATH provided by the image without changing any of the other environment variables.

mounts

[[directives]]
mounts = [
    { fs_type = "tmp", mount_point = "/tmp" },
    { fs_type = "proc", mount_point = "/proc" },
    { fs_type = "sys", mount_point = "/sys" },
]

This field sets the mounts field of the job spec. It must be a list of tables, each of which must have two fields:

• fs_type: This indicates the type of special file system to mount, and must be one of the following strings: "tmp", "proc", or "sys".
• mount_point: This must be a string. It specifies the mount point within the container for the file system.

added_mounts

This field is like mounts, except it appends to the job spec's mounts field instead of replacing it.

devices

[[directives]]
devices = ["fuse", "full", "null", "random", "tty", "urandom", "zero"]

This field sets the devices field of the job spec. It must be a list of strings, whose elements must be one of:

• "full"
• "fuse"
• "null"
• "random"
• "tty"
• "urandom"
• "zero"

The order of the values doesn't matter, as the list is treated like an unordered set.

added_devices

This field is like devices, except it inserts into to the job spec's devices set instead of replacing it.

working_directory

[[directives]]
working_directory = "/home/root/"

This field sets the working_directory field of the job spec. It must be a string.

This field can't be set in the same directive as image if the image.use contains "working_directory".

enable_loopback

[[directives]]
enable_loopback = true

This field sets the enable_loopback field of the job spec. It must be a boolean.

enable_writable_file_system

[[directives]]
enable_writable_file_system = true

This field sets the enable_writable_file_system field of the job spec. It must be a boolean.

user

[[directives]]
user = 1000

This field sets the user field of the job spec. It must be an unsigned, 32-bit integer.

group

[[directives]]
group = 1000

This field sets the group field of the job spec. It must be an unsigned, 32-bit integer.

timeout

[[directives]]
timeout = 60

This field sets the timeout field of the job spec. It must be an unsigned, 32-bit integer.

Files in Target Directory

cargo-maelstrom stores a number of files in the workspace's target directory. This chapter lists them and explains what they're for.

Except in the case of the local worker, cargo-maelstrom doesn't currently make any effort to clean up these files.

cargo-maelstrom also uses the container-images cache. That cache is not stored in the target directory, as it can be reused by different Maelstrom clients.

Local Worker

When run in standalone mode, the local worker stores its files in maelstrom-local-worker/ in the target directory. The cache-size configuration value indicates the target size of this cache directory.

Manifest Files

cargo-maelstrom uses "manifest files" for non-tar layers. These are like tar files, but without the actual data contents. These files are stored in maelstrom-manifests/ in the target directory.

Client Log File

The local client process — the one that cargo-maelstrom talks to, and that contains the local worker — has a log file that is stored at maelstrom-client-process.log in the target directory.

Test Listing

When cargo-maelstrom finishes, it updates a list of all of the tests in the workspace. This is used to predict the amount of tests will be run in subsequent invocations. This is stored in the maelstrom-test-listing.toml file in the target directory.

File Digests

Files uploaded to the broker are identified by a hash of their file contents. Calculating these hashes can be time consuming so cargo-maelstrom caches this information. This is stored in maelstrom-cached-digests.toml in the target directory.

Configuration Values

cargo-maelstrom supports the following configuration values:

broker

The broker configuration value specifies the socket address of the broker. This configuration value is optional. If not provided, cargo-maelstrom will run in standalone mode.

Here are some example value socket addresses:

• broker.example.org:1234
• 192.0.2.3:1234
• [2001:db8::3]:1234

log-level

See here.

cargo-maelstrom always prints log messages to stdout.

quiet

The quiet configuration values, if set to true, causes cargo-maelstrom to be more more succinct with its output. If cargo-maelstrom is outputting to a terminal, it will display a single-line progress bar indicating all test state, then print a summary at the end. If not outputting to a terminal, it will only print a summary at the end.

timeout

The optional timeout configuration value provides the timeout value to use for all tests. This will override any value set in maelstrom-test.toml.

cache-size

This is a local-worker setting. See here for more.

inline-limit

This is a local-worker setting. See here for more.

slots

This is a local-worker setting. See here for more.

Cargo Settings

cargo-maelstrom shells out to cargo to get metadata about tests and to build the test artifacts. For the former, it uses cargo metadata. For the latter, it uses cargo test --no-run.

cargo-maelstrom supports a number of command-line options that are passed through directly to cargo. It does not inspect these values at all.

cargo-maelstrom doesn't accept multiple instances of the --features command-line option. Instead, combine the features into a single, comma-separated argument like this: --features=feat1,feat2,feat3.

cargo-maelstrom doesn't accept the --release alias. Use --profile=release instead.

Command-Line Options

Besides the standard command-line options and the options for configuration values, cargo-maelstrom supports additional command-line-options.

--init

The --init command-line option is used to create a starter maelstrom-test.toml file. See here for more information.

--list-tests or --list

The --list-tests (or --list) command-line option causes cargo-maelstrom to build all required test binaries, then print the tests that would normally be run, without actually running them.

This option can be combined with --include and --exclude.

--list-binaries

The --list-binaries command-line option causes cargo-maelstrom to print the names and types of the crates that it would run tests from, without actually building any binaries or running any tests.

This option can be combined with --include and --exclude.

--list-packages

The --list-packages command-line option causes cargo-maelstrom to print the packages from which it would run tests, without actually building any binaries or running any tests.

This option can be combined with --include and --exclude.

--include and --exclude

The --include (-i) and --exclude (-x) command-line options control which tests cargo-maelstrom runs or lists.

These options take a test filter pattern. The --include option includes any test that matches the pattern. Similarly, --exclude pattern excludes any test that matches the pattern. Both options are allowed to be repeated arbitrarily.

The tests that are selected are the set which match any --include pattern but don't match any --exclude pattern. In other words, --excludes have precedence over --includes, regardless of the order they are specified.

If no --include option is provided, cargo-maelstrom acts as if an --include all option was provided.

Working with Workspaces

When you specify a filter with a package, cargo-maelstrom will only build the matching packages. This can be a useful tip to remember when trying to run a single test.

If we were to run something like:

cargo maelstrom --include "name.equals(foobar)"

cargo-maelstrom would run any test which has the name "foobar". A test with this name could be found in any of the packages in the workspace, so it is forced to build all of them. But if we happened to know that only one package has this test — the baz package — it would be faster to instead run:

cargo maelstrom --include "package.equals(baz) && name.equals(foobar)"

Since we specified that we only care about the baz package, cargo-maelstrom will only bother to build that package.

Abbreviations

As discussed here, unambiguous prefixes can be used in patterns. This can come in handy when doing one-offs on the command line. For example, the example above could be written like this instead:

cargo maelstrom -i 'p.eq(baz) & n.eq(foobar)'

maelstrom-broker

The maelstrom-broker is the coordinator and scheduler for a Mealstrom cluster. In order to have a cluster, there must be a broker. The broker must be started before clients and workers, as the clients and workers connect to the broker, and will exit if they can't establish a connection.

The broker doesn't consume much CPU, so it can be run on any machine, including a worker machine. Ideally, whatever machine it runs on should have good throughput with the clients and workers, as all artifacts are first transferred from the clients to the broker, and then from the broker to workers.

Clients can be run in standalone mode where they don't need access to a cluster. In that case, there is no need to run a broker.

Cache

The broker maintains a cache of artifacts that are used as file system layers for jobs. If a client submits a job, and there are required artifacts for the job that the broker doesn't have in its cache, it will ask the client to transfer them. Later, when the broker submits the job to a worker, the worker may turn around and request missing artifacts from the broker.

A lot of artifacts are reused between jobs, and also between client invocations. So, the larger the broker's cache, the better. Ideally, it should be at least a few multiples of the working set size.

Command-Line Options

maelstrom-broker supports the standard command-line options, as well as a number of configuration values, which are covered in the next chapter.

Configuration Values

maelstrom-broker supports the following configuration values:

log-level

See here.

The broker always prints log messages to stderr.

cache-root

The cache-root configuration value specifies where the cache data will go. It defaults to $XDG_CACHE_HOME/maelstrom/broker, or ~/.cache/maelstrom/broker if XDG_CACHE_HOME isn't set. See the XDG spec for information.

cache-size

The cache-size configuration value specifies a target size for the cache. Its default value is 1 GB. When the cache consumes more than this amount of space, the broker will remove unused cache entries until the size is below this value.

It's important to note that this isn't a hard limit, and the broker will go above this amount in two cases. First, the broker always needs all of the currently-executing jobs' layers in cache. Second, the broker currently first downloads an artifact from the client in its entirety, then adds it to the cache, then removes old values if the cache has grown too large. In this scenario, the combined size of the downloading artifact and the cache may exceed cache-size.

For these reasons, it's important to leave some wiggle room in the cache-size setting.

port

The port configuration value specifies the port the broker will listen on for connections from clients and workers. It must be an integer value in the range 0–65535. A value of 0 indicates that the operating system should choose an unused port. The broker will always listen on all IP addresses of the host.

http-port

the http-port configuration value specifies the port the broker will serve the web UI on. A value of 0 indicates that the operating system should choose an unused port. The broker will always listen on all IP addresses of the host.

Running as systemd Service

You may choose to run maelstrom-broker in the background as a systemd service. This chapter covers one way to do that.

The maelstrom-broker does not need to run as root. Given this, we can create a non-privileged user to run the service:

sudo adduser --disabled-login --gecos "Maelstrom Broker User" maelstrom-broker
sudo -u maelstrom-broker mkdir ~maelstrom-broker/cache
sudo -u maelstrom-broker touch ~maelstrom-broker/config.toml
sudo cp ~/.cargo/bin/maelstrom-broker ~maelstrom-broker/

This assumes the maelstrom-broker binary is installed in ~/.cargo/bin/.

Next, create a service file at /etc/systemd/system/maelstrom-broker.service and fill it with the following contents:

[Unit]
Description=Maelstrom Broker

[Service]
User=maelstrom-broker
WorkingDirectory=/home/maelstrom-broker
ExecStart=/home/maelstrom-broker/maelstrom-broker \
    --config-file /home/maelstrom-broker/config.toml
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

Next, edit the file at /home/maelstrom-broker/config.toml and fill it with the following contents:

port = 9000
http-port = 9001
cache-root = "/home/maelstrom-broker/cache"

You can add other configuration values as you please.

Finally, enable and start the broker:

sudo systemctl enable maelstrom-broker
sudo systemctl start maelstrom-broker

The broker should be running now. If you want, you can verify this by attempting to pull up the web UI, or by verifying the logs messages with journalctl.

Web UI

The broker has a web UI that is available by connecting via the configured HTTP port.

The following is an explanation of the various elements on the web UI.

Connected Machines

The web UI contains information about the number of client and the number of worker connected to the broker. The web UI itself is counted as a client.

Slots

Each running job consumes one slot. The more workers connected, the more slots are available. The broker shows the total number of available slots as well as the number currently used.

Job Statistics

The web UI contains information about current and past jobs. This includes the current number of jobs and graphs containing historical information about jobs and their states. There is a graph per connected client as well as an aggregate graph at the top. The graphs are all stacked line-charts. See Job States for information about what the various states mean.

maelstrom-worker

The maelstrom-worker is used to execute jobs in a Maelstrom cluster. In order to do any work, a cluster must have at least one worker.

The system is designed to require only one worker per node in the cluster. The worker will then run as many jobs in parallel as it has "slots". By default, it will have one slot per CPU, but it can be configured otherwise.

Clients can be run in standalone mode where they don't need access to a cluster. In that case, they will have a internal, local copy of the worker.

All jobs are run inside of containers. In addition to providing isolation to the jobs, this provides some amount of security for the worker.

Cache

Each job requires a file system for its containers. The worker provides these file systems via FUSE. It keeps the artifacts necessary to implement these file systems in its cache directory. Artifacts are reused if possible.

The worker will strive to keep the size of the cache under the configurable limit. It's important to size the cache properly. Ideally, it should be a small multiple larger than the largest working set.

Command-Line Options

maelstrom-worker supports the standard command-line options, as well as a number of configuration values, which are covered in the next chapter.

Configuration Values

maelstrom-worker supports the following configuration values:

broker

The broker configuration value specifies the socket address of the broker. This configuration value must be provided. The worker will exit if it fails to connect to the broker, or when its connection to the broker terminates.

Here are some example value socket addresses:

• broker.example.org:1234
• 192.0.2.3:1234
• [2001:db8::3]:1234

log-level

See here.

The worker always prints log messages to stderr.

cache-root

The cache-root configuration value specifies where the cache data will go. It defaults to $XDG_CACHE_HOME/maelstrom/worker, or ~/.cache/maelstrom/worker if XDG_CACHE_HOME isn't set. See the XDG spec for information.

cache-size

The cache-size configuration value specifies a target size for the cache. Its default value is 1 GB. When the cache consumes more than this amount of space, the worker will remove unused cache entries until the size is below this value.

It's important to note that this isn't a hard limit, and the worker will go above this amount in two cases. First, the worker always needs all of the currently-executing jobs' layers in cache. Second, the worker currently first downloads an artifact in its entirety, then adds it to the cache, then removes old values if the cache has grown too large. In this scenario, the combined size of the downloading artiface and the cache may exceed cache-size.

For these reasons, it's important to leave some wiggle room in the cache-size setting.

inline-limit

The inline-limit configuration value specifies how many bytes of stdout or stderr will be captured from jobs. Its default value is 1 MB. If stdout or stderr grows larger, the client will be given inline-limit bytes and told that the rest of the data was truncated.

In the future we will add support for the worker storing all of stdout and stderr if they exceed inline-limit. The client would then be able to download it "out of band".

slots

The slots configuration value specifies how many jobs the worker will run concurrently. Its default value is the number of CPU cores on the machine. In the future, we will add support for jobs consuming more than one slot.

Running as systemd Service

You may choose to run maelstrom-worker in the background as a systemd service. This chapter covers one way to do that.

The maelstrom-worker does not need to run as root. Given this, we can create a non-privileged user to run the service:

sudo adduser --disabled-login --gecos "Maelstrom Worker User" maelstrom-worker
sudo -u maelstrom-worker mkdir ~maelstrom-worker/cache
sudo -u maelstrom-worker touch ~maelstrom-worker/config.toml
sudo cp ~/.cargo/bin/maelstrom-worker ~maelstrom-worker/

This assumes the maelstrom-worker binary is installed in ~/.cargo/bin/.

Next, create a service file at /etc/systemd/system/maelstrom-worker.service and fill it with the following contents:

[Unit]
Description=Maelstrom Worker

[Service]
User=maelstrom-worker
WorkingDirectory=/home/maelstrom-worker
ExecStart=/home/maelstrom-worker/maelstrom-worker \
    --config-file /home/maelstrom-worker/config.toml
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

Next, edit the file at /home/maelstrom-worker/config.toml and fill it with the following contents:

broker = "<broker-machine-address>:<broker-port>"
cache-root = "/home/maelstrom-worker/cache"

The <broker-machine-address> and <broker-port> need to be substituted with their actual values. You can add other configuration values as you please.

Finally, enable and start the worker:

sudo systemctl enable maelstrom-worker
sudo systemctl start maelstrom-worker

The worker should be running now. If you want, you can verify this by pulling up the broker web UI and checking the worker count, or by looking at the broker's log messages.

maelstrom-run

maelstrom-run is a program for running arbitrary commands on a Maelstrom cluster.

Command-Line Options

The maelstrom-run supports the standard command-line options, as well as a number of configuration values, which are covered in the next chapter.

Job Specification

This chapter hasn't been filled in yet.

Configuration Values

maelstrom-run supports the following configuration values:

broker

The broker configuration value specifies the socket address of the broker. This configuration value is optional. If not provided, maelstrom-run will run in standalone mode.

Here are some example value socket addresses:

• broker.example.org:1234
• 192.0.2.3:1234
• [2001:db8::3]:1234

log-level

See here.

maelstrom-run always prints log messages to stderr.

cache-size

This is a local-worker setting. See here for more.

inline-limit

This is a local-worker setting. See here for more.

slots

This is a local-worker setting. See here for more.