Note that the to_json method prefers the taskgraph's dependencies information
(edges) to that from the task.dependencies entries. At a few points in
task-graph generation, these values differ, although that is expected (for
example, the full task set contains no edges, but that information is still in
task.dependencies). Unifying that representation leads to some difficulty with
task transforms that reach into the dependency tree (beetmover), so the
different representations are left as-is.
MozReview-Commit-ID: GeW8HNwFA9Z
Considering docker images contents depend very much on the moment they
were built, it is possible that two images with the same hash in the
taskcluster index (at different levels) have different contents. When
this happens, the build or test results could be significantly
different between e.g. try and mozilla-central, possibly leading to
misleading results at landing time.
So if for some reason multiple levels have images for the same hash, the
one used at the highest level should be prefered, such that try uses the
same as mozilla-central once mozilla-central generates the image for
that hash, even if there is an image previously generated for try.
* Compress docker images with zstd
* Removed need for context.tar from decision task
* Index images by level rather than project
MozReview-Commit-ID: 4RL4QXNWmpd
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
