Test Filter Patterns

There are times when a user needs to concisely specify a set of tests to cargo-maelstrom. One of those is on the command line: cargo-maelstrom can be told to only run a certain set of tests, or to exclude some tests. Another is the filter field of cargo-maelstrom.toml directives. This is used to choose which tests a directive applies too.

In order to allow users to easily specify a set of tests to cargo-maelstrom, we created the domain-specific pattern language described here.

If you are a fan of formal explanations check out the BNF. Otherwise, this page will attempt to give a more informal explanation of the language.

Simple Selectors

The most basic patterns are "simple selectors". These are only sometimes useful on their own, but they become more powerful when combined with other patterns. Simple selectors consist solely of one of the these identifiers:

Simple SelectorWhat it Matches
true, any, allany test
false, noneno test
libraryany test in a library crate
binaryany test in a binary crate
benchmarkany test in a benchmark crate
exampleany test in an example crate
testany test in a test crate

Simple selectors can optionally be followed by (). That is, library() and library are equivalent patterns.

Compound Selectors

"Compound selector patterns" are patterns like package.equals(foo). They combine "compound selectors" with "matchers" and "arguments". In our example, package is the compound selector, equals is the matcher, and foo is the argument.

These are the possible compound selectors:

Compound SelectorSelected Name
namethe name of the test
packagethe name of the test's package
binarythe name of the test's binary target
benchmarkthe name of the test's benchmark target
examplethe name of the test's example target
testthe name of the test's (integration) test target

Documentation on the various types of targets in cargo can be found here.

These are the possible matchers:

MatcherMatches If Selected Name...
equalsexactly equals argument
containscontains argument
starts_withstarts with argument
ends_withends with argument
matchesmatches argument evaluated as regular expression
globsmatches argument evaluated as glob pattern

Compound selectors and matchers are separated by . characters. Arguments are contained within delimiters, which must be a matched pair:

LeftRight
()
[]
{}
<>
//

The compound selectors binary, benchmark, example, and test will only match if the test is from a target of the specified type and the target's name matches. In other words, binary.equals(foo) can be thought of as shorthand for the compound pattern (binary && binary.equals(foo)).

Let's put this all together with some examples:

PatternWhat it Matches
name.equals(foo::tests::my_test)Any test named "foo::tests::my_test".
binary.contains/maelstrom/Any test in a binary crate, where the executable's name contains the substring "maelstrom".
package.matches{(foo)*bar}Any test whose package name matches the regular expression (foo)*bar.

Compound Expressions

Selectors can be joined together with operators to create compound expressions. These operators are:

OperatorsAction
!, ~, notLogical Not
&, &&, andLogical And
|, ||, orLogical Or
\, -, minusLogical Difference
(, )Grouping

The "logical difference" action is defined as follows: A - B == A && !B.

As an example, to select tests named foo or bar in package baz:

(name.equals(foo) || name.equals(bar)) && package.equals(baz)

As another example, to select tests named bar in package baz or tests named foo from any package:

name.equals(foo) || (name.equals(bar) && package.equals(baz))

Abbreviations

Selector and matcher names can be shortened to any unambiguous prefix.

For example, the following are all the same

name.equals(foo)
name.eq(foo)
n.eq(foo)

We can abbreviate name to n since no other selector starts with "n", but we can't abbreviate equals to e because there is another selector, ends_with, that also starts with an "e".