Container Images

Maelstrom runs each job in its own container. By default, these containers are minimal and are built entirely from files copied from the local machine. This works well for compiled languages (like Rust), where job don't depend on executing other programs. However, for interpreted languages (like Python), or in situations where the job needs to execute other programs, Maelstrom provides a way to create containers based off of a standard, OCI container images. These container images can be retrieved from a container registry like Docker Hub, or provided from the local machine.

Container Image URIs

A container image is specified using the URI format defined by the Containers project. In particular, here is their specification of the URI format. We currently support the following URI schemes.

docker://docker_reference

This scheme indicates that the container image should be retrieved from a container registry using the Docker Registry HTTP API V2. The container registry, container name, and optional tags are specified in docker_reference.

docker-reference has the format: name[:tag | @digest]. If name does not contain a slash, it is treated as docker.io/library/name. Otherwise, the component before the first slash is checked to see if it is recognized as a hostname[:port] (i.e., it contains either a . or a :, or the component is exactly localhost). If the first component of name is not recognized as a hostname[:port], name is treated as docker.io/name.

Here is how the Containers project specifies this scheme.

oci:/path[:reference]

This scheme indicates that the container images should be retrieved from a local directory at path in the format specified by the Open Container Image Layout Specification.

Any characters after the first : are considered to be part of reference, which is used to match an org.opencontainers.image.ref.name annotation in the top-level index. If reference is not specified when reading an archive, the archive must contain exactly one image.

Here is how the Containers project specifies this scheme.

oci-archive:/path[:reference]

This scheme indicates that the container images should be retrieved from a tar file at path with contents in the format specified by the Open Container Image Layout Specification.

Any characters after the first : are considered to be part of reference, which is used to match an org.opencontainers.image.ref.name annotation in the top-level index. If reference is not specified when reading an archive, the archive must contain exactly one image.

Here is how the Containers project specifies this scheme.

Cached Container Images

When a container image is specified, the client will first download or copy the image into its cache directory. It will then use the internal bits of the OCI image — most importantly the file-system layers — to create the container for the job.

The cache directory can be set with the container-image-depot-root configuration value. If this value isn't set, but XDG_CACHE_HOME is set and non-empty, then $XDG_CACHE_HOME/maelstrom/containers will be used. Otherwise, ~/.cache/maelstrom/containers will be used. See the XDG Base Directories specification for more information.

Lock File

When a client first resolves a container registry tag, it stores the result in a local lock file. Subsequently, it will use the exact image specified in the lock file instead of resolving the tag again. This guarantees that subsequent runs use the same images as previous runs.

The lock file is maelstrom-container-tags.lock, stored in the project directory. It is recommended that this file be committed to revision control, so that others in the project, and CI, use the same images when running tests.

To update a tag to the latest version, remove the corresponding line from the lock file and then run the client. This will force it to re-evaluate the tag and store the new results.

Authentication

When a client connects to a container registry, it may need to authenticate. Maelstrom currently only supports registries for no authentication or with anonymous authentication. Password authentication will be added in the future.

Image Registry TLS Certificates

Maelstrom always uses HTTPS to connect to container registries. By default, it will reject self-signed or otherwise invalid certificates. However, clients can be told to accept these certificates with the accept-invalid-remote-container-tls-certs configuration value.

Configuration Values

All clients support the following container-image-related configuration values: