In this case, the tasks must have the same schedulerId as the existing task,
so this is calculated using parameters['level'].
MozReview-Commit-ID: G8EE2kvFstT
This uses the run_on_projects attribute introduced earlier for most branches,
adjusts the `ash` method to handle that branch as the legacy implementation
did, and updates try syntax to match builds as well as tests.
In the process, this enables optimizing target tasks, meaning that tasks
specifically requested in the try syntax might be optimized. While this is
probably not ideal, it matches the existing behavior of try (where `-j all` is
the default but all jobs are set to run only when certain files have been
modified). This change can be reverted later, in a more advanced version of
try.
MozReview-Commit-ID: 5FYeUTAsafr
MikeLing initially did this in bug 1287018. The intent of this conditional was
to make optimization faster by not even checking most tasks, based on the
assumption that if the prerequisite to a task has changed (for example, a
docker image or a build), then naturally we will want to execute that task.
However, as we have developed actual optimization methods, this has proven not
to be the case: we might want to optimize a test out if its inputs have not
changed, even if a new installer has been built. Similarly, SETA may optimize
tasks out even if their inputs have changed.
MozReview-Commit-ID: LgHET3Z84GB
Without this, current umask may influence test results. That was
causing differences between automation and local runs.
MozReview-Commit-ID: 1eu613aBpKB
Use the source RIDEALONG_BUILDS value in the module under test so that changes
to that variable do not cause the test to fail.
MozReview-Commit-ID: EfHQ7baBziB
The function was only used once and was providing little to no value.
A test of this function has been removed. Tests for the lower-level
context creation function are sufficient.
MozReview-Commit-ID: D9EhmZQlqG5
A limitation of traditional docker build context generation is it
only includes files from the same directory as the Dockerfile. When
repositories have multiple, related Dockerfiles, this limitation
results file duplication or putting all Dockerfiles in the same
directory (which isn't feasible for mozilla-central since they would
need to be in the root directory).
This commit enhances Dockerfiles to allow *any* file from the
repository checkout to be ADDed to the docker build context.
Using the syntax "# %include <path>" you are able to include paths
or directories (relative from the top source directory root) in the
generated context archive. Files add this way are available under the
"topsrcdir/" path and can be ADDed to Docker images.
Since context archive generation is deterministic and the hash of
the resulting archive is used to determine when images need to be
rebuilt, any extra included file that changes will change the hash
of the context archive and force image regeneration.
Basic tests for the new feature have been added.
MozReview-Commit-ID: 4hPZesJuGQV
This restores order to only having a single hash for a context
directory.
Using a tempfile here is a bit unfortunate. It can be optimized later,
if needed.
MozReview-Commit-ID: LMNsvt3fDYx
Now that the context tar creation function is standalone and doesn't
rely on external state, we can start unit testing it easier.
We establish a basic unit test that verifies the function works as
advertised and that output is deterministic.
MozReview-Commit-ID: H4MY28PiHSN
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
This enables kinds that generate tasks based on those output by another kind.
For example, the test kind might generate a set of test tasks for each build
task.
MozReview-Commit-ID: K7ha9OmJ6gd
This enables kinds that generate tasks based on those output by another kind.
For example, the test kind might generate a set of test tasks for each build
task.
MozReview-Commit-ID: K7ha9OmJ6gd
* Implement & document optimization (although legacy kind doesn't do much of it)
* Introduce `optimize_target_tasks` parameter to control whether tasks in the
target set can be optimized (no for try, yes for most other branches)
* Refactor to include resolved taskIds in the optimized task graph
* Include a `label-to-taskid.json` artifact.
* Introduce {'task-reference': '... <dependency-name> ...'} for referring to
parent tasks' taskId.
MozReview-Commit-ID: LWvlWNz49U5
This adds support for the `-t`/`--talos` option, matching such jobs against
`talos_try_name`. There are no such tasks just yet.
MozReview-Commit-ID: FTEx7Nyyi9Z
The `taskgraph` package generates TaskCluster task graphs based on collections
of task "kinds". Initially, there is only one kind, the "legacy" kind, which
reads the YAML files from `testing/taskcluster/tasks` to generate the task
graph.
Try syntax is implemented by filtering the tasks in the taskgraph after it has
been created, then extending the result to include any prerequisite tasks.
A collection of `mach taskgraph` subcommands are provided for developers to
extend or debug the task-graph generation process.
MozReview-Commit-ID: 1TJCns4XxZ8
