We add the following command line options to Taskcluster try syntax:
--spsProfile: enable profile mode.
--rebuild-talos <N>: retrigger talos tests N times.
--setenv <VAR>=<val>: add extra environments variables.
--tag <TAG>: run tests only the tag TAG.
--no-retry: doesn't retry failed jobs.
We have a chicken-egg problem, as we first generate the full task graph
and then parse the try message. But the graph generation step needs to
know the try message to process the aforementioned options. The
solution is to parse the message before graph generation and then
pass the command line options to the transforms. Then, each transform
can look at the option that interests it and process it accordingly.
The message parse function is configured in kind.yml, which gives some
flexibility for future implementations of alternative syntaxes.
MozReview-Commit-ID: EQlE6q5E8z7
While we don't have physical machines running taskcluster-worker, we
want to schedule talos tests on Taskcluster and run them on buildbot.
buildbot-bridge acts as a worker that runs its tasks in buildbot
machines. Any talos task is redirected to buildbot-bridge.
MozReview-Commit-ID: Im9IhhT0Mr1
This commit introduces support for performing build and test variations
with Stylo enabled.
From a high level, this is similar to the e10s/non-e10s variations that
we currently run. However, it is different in that Stylo is a build
time (not run time) flag. So, we need to perform separate builds for
Stylo.
In the current implementation of the task graph, we can either
introduce a new platform or a new build type/configuration. I've
chosen to introduce a new platform if nothing else so the Treeherder
results clearly separate Stylo from non-Stylo. This decision can be
revisited in the future once Stylo becomes more stable and visible
outside the "stylo" repository.
Stylo functionality is limited to layout functionality and Stylo
developers aren't very interested in many tests not related to layout
at the moment. So we introduce a new test set running a limited
functionality of tests in Stylo builds.
In the current state of this commit, Stylo builds will permafail.
This is because Stylo builds require a vendored copy of Servo. Logic
for pruning the task graph of Stylo/Servo related tasks will be
implemented in subsequent commits.
MozReview-Commit-ID: JivhkpXDjIw
Job tasks are tasks that are not tied to a specific build or test. As such, they cannot be scheduled
with the regular -p/-u try syntax options. There exists a -j try syntax option, to schedule them, which
defaults to running "all" of them if not specified.
However, there is a bug here where they will only default to "all" if a try syntax exists in the commit
message. They will not be considered if a developer pushes to try without a try syntax. This happens
because self.jobs is initially initialized with '[]' and we use None to determine when to schedule "all"
later on.
I want to move towards a world without try syntax, so we should start improving the UX of the no try
syntax use case.
Note: When I say "schedule" here, I mean added to the target set. They may still be optimized away.
MozReview-Commit-ID: 4TrC84RGiaL
This patch prevents tests which have the 'run_on_projects' attribute assigned to an empty set from running when the try message contains '-p all' and '-u all' together. It also makes the 'match_test' function a little more readable and updates the 'attributes.rst' document to reflect the changes that were made.
MozReview-Commit-ID: IMk0cmSza8U
If you want to run tests on Linux32 and Linux64, you use trychooser syntax like
"-u all[Ubuntu]" which taskcluster supports, but if you also built ASan, and
want tests on ASan, there's no supported documented way to get them. That sounds
like an edge-case, except that to get tests to run on WinXP or Win8 you have to
explicitly list them, so to get tests on every platform you build you have to
list all of them, and if ASan is one of them you have to somehow know to dig
around in the .yml and to know that linux64-asan is the thing to dig out.
MozReview-Commit-ID: 2REf0cUWmK8
The trychooser website advertises 'Android 4.3' as the alias to choose Android
tests in [] syntax, but without support for that in taskcluster, someone who
wants both opt and debug tests would need to guess, with no support or
documentation, that they should use [Android 4.3,android-4.3-arm7-api-15]
MozReview-Commit-ID: 4rEKcvqAfDH
This uses the run_on_projects attribute introduced earlier for most branches,
adjusts the `ash` method to handle that branch as the legacy implementation
did, and updates try syntax to match builds as well as tests.
In the process, this enables optimizing target tasks, meaning that tasks
specifically requested in the try syntax might be optimized. While this is
probably not ideal, it matches the existing behavior of try (where `-j all` is
the default but all jobs are set to run only when certain files have been
modified). This change can be reverted later, in a more advanced version of
try.
MozReview-Commit-ID: 5FYeUTAsafr
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
This adds support for the `-t`/`--talos` option, matching such jobs against
`talos_try_name`. There are no such tasks just yet.
MozReview-Commit-ID: FTEx7Nyyi9Z
The `taskgraph` package generates TaskCluster task graphs based on collections
of task "kinds". Initially, there is only one kind, the "legacy" kind, which
reads the YAML files from `testing/taskcluster/tasks` to generate the task
graph.
Try syntax is implemented by filtering the tasks in the taskgraph after it has
been created, then extending the result to include any prerequisite tasks.
A collection of `mach taskgraph` subcommands are provided for developers to
extend or debug the task-graph generation process.
MozReview-Commit-ID: 1TJCns4XxZ8
