Directives
Directives are the other type of elements in maelstrom-pytest.toml
, besides
containers
.
The purpose of directives is to map individual tests to their corresponding settings, that is, job specifications. As there may be thousands of tests in a single project, we need a system that allows for providing settings for a broad swath of tests, while at the same time allowing for individual configuration when required. Directives is how we chose to solve the problem.
Every directive has a filter: a pattern that tests either match against or
don’t. To arrive at the job spec for a job, maelstrom-pytest
starts with a
default candidate job spec, then inspects each directive in order. If the
directive’s filter matches the test, then the directive is applied to the
candidate job spec. The process does not stop after a single match, but instead
proceeds through all directives, in order.
filter
[[directives]]
filter = "package.equals(maelstrom) && name.equals(io::splicer)"
image.name = "docker://python"
image.use = ["layers", "environment"]
This field must be a string, which is interpreted as a test filter
pattern. The directive only applies to tests that match the
filter. If there is no filter
field, the directive applies to all tests.
Sometimes it is useful to use multi-line strings for long patterns:
[[directives]]
filter = """
package.equals(maelstrom-client) ||
package.equals(maelstrom-client-process) ||
package.equals(maelstrom-container) ||
package.equals(maelstrom-fuse) ||
package.equals(maelstrom-util)"""
layers = [{ stubs = ["/tmp/"] }]
mounts = [{ type = "tmp", mount_point = "/tmp" }]
A directive without an explicit filter matches every test.
ignore
[[directives]]
filter = "name.contains(broken_test)"
ignore = true
This field specifies that any tests matching the directive should not be run.
maelstrom-pytest
prints ignored tests, with a special “ignored” state. When
tests are listed, ignored tests are listed normally.
timeout
[[directives]]
timeout = 60
This field sets the timeout
field of the
job spec. It must be an unsigned, 32-bit integer.
Setting the timeout to 0
results in no timeout.
By default, tests don’t have a timeout.
There is also a timeout
configuration value. If set
set, the configuration value overrides the directive’s field.
Container Specification Fields
Each job spec has an embedded container spec. As a result, all of the
containers
fields are also valid in directives. However,
things can get tricky with the inheritance of container specs as well as the
application of directives. We’ve decided on a set of rules that we hope makes
thing powerful but yet understandable.
Setting parent
or image
When a directive specifies the parent
or
image
field, then all other container spec fields
are reset. For example:
[[directives]]
user = 42
added_layers = [{ stubs = ["/tmp/"] }]
added_mounts = [{ type = "tmp", mount_point = "/tmp" }]
timeout = 10
[[directives]]
filter = "package.equals(maelstrom) && name.equals(io::splicer)"
image = "docker://python"
In this case, any test that matches the second directive will have its
layers
, mounts
, and user
reset. However, it would retain the timeout
,
since timeout
isn’t a container spec field.
When either parent
or image
are provided, then all of the container spec
fields have their same meanings and restrictions as they do
when specifying a named container. In particular, setting vector fields will
act just as described here.
Without parent
or image
A directive can still set container spec fields even if it doesn’t include the
parent
or image
fields. However, in this case, the interpretation of vector
fields is different.
The added_layers
,
added_environment
, or
added_mounts
are
always allowed in this case. They will append to relevant container spec field,
regardless of whether the container spec inherits the field from a parent.
Similarly, the layers
,
environment
, and
mounts
are always allowed as well. They will
overwrite the relevant container spec field, without changing the how the
container spec inherits the field from a parent. This last part will probably
change. It’s likely that in an upcoming release we will change the behavior
in this edge case so that container spec will also be modified so that it
doesn’t inherit the relevant field from its parent.