Note that the to_json method prefers the taskgraph's dependencies information
(edges) to that from the task.dependencies entries. At a few points in
task-graph generation, these values differ, although that is expected (for
example, the full task set contains no edges, but that information is still in
task.dependencies). Unifying that representation leads to some difficulty with
task transforms that reach into the dependency tree (beetmover), so the
different representations are left as-is.
MozReview-Commit-ID: GeW8HNwFA9Z
Now that we use the real geckolib and have all dependencies vendored,
the dummy geckolib is no longer required, so we remove it.
Also, the taskgraph code for testing for Servo's presence always
passes and is no longer needed, so we remove it.
Pushed on a CLOSED TREE because ¯\_(ツ)_/¯
MozReview-Commit-ID: ITAqArK4Bks
I want to get Servo vendored into servo/. The previous plan was to
replace the dummy geckolib with the real deal when the vendoring is
done. Unfortunately, this will require a significant `cargo vendor`
change, which we want to punt on for a bit.
So, this commit moves our dummy geckolib outside of servo/ so we
don't need to `cargo update` or `cargo vendor` when the real servo/
is installed.
The change to toolkit/library/rust/shared/Cargo.toml can be reverted
in the stylo repo to allow it to use the real geckolib.
We also update the taskgraph code for detecting Servo. Previously,
it looked for a file in the possibly-vendored servo/ directory. Once
the vendoring happens, this check will always pass. But without the
real geckolib, the Servo builds will fail. So, we change the check
to look for the real geckolib. This is implemented a bit hackily.
But it will be short-lived until we run `cargo vendor`.
MozReview-Commit-ID: CxGTwy6bK9j
We add the following command line options to Taskcluster try syntax:
--spsProfile - enable profile mode.
--rebuild-talos <N> - retrigger talos tests N times.
--setenv <VAR>=<val> - add extra environments variables.
--tag <TAG> - run tests only the tag TAG.
--no-retry - doesn't retry failed jobs.
We have a chicken-egg problem, as we first generate the full task graph
and then parse the try message. But the graph generation step needs to
know the try message to process the aforementioned options. The
solution is to parse the message before graph generation and then
pass the command line options to the transforms. Then, each transform
can look at the option that interests it and process it accordingly.
The message parse function is configured in kind.yml, which gives some
flexibility for future implementations of alternative syntaxes.
MozReview-Commit-ID: GPFdi0FD6Vn
We add the following command line options to Taskcluster try syntax:
--spsProfile - enable profile mode.
--rebuild-talos <N> - retrigger talos tests N times.
--setenv <VAR>=<val> - add extra environments variables.
--tag <TAG> - run tests only the tag TAG.
--no-retry - doesn't retry failed jobs.
We have a chicken-egg problem, as we first generate the full task graph
and then parse the try message. But the graph generation step needs to
know the try message to process the aforementioned options. The
solution is to parse the message before graph generation and then
pass the command line options to the transforms. Then, each transform
can look at the option that interests it and process it accordingly.
The message parse function is configured in kind.yml, which gives some
flexibility for future implementations of alternative syntaxes.
MozReview-Commit-ID: DMwRjuV2vpf
We add the following command line options to Taskcluster try syntax:
--spsProfile: enable profile mode.
--rebuild-talos <N>: retrigger talos tests N times.
--setenv <VAR>=<val>: add extra environments variables.
--tag <TAG>: run tests only the tag TAG.
--no-retry: doesn't retry failed jobs.
We have a chicken-egg problem, as we first generate the full task graph
and then parse the try message. But the graph generation step needs to
know the try message to process the aforementioned options. The
solution is to parse the message before graph generation and then
pass the command line options to the transforms. Then, each transform
can look at the option that interests it and process it accordingly.
The message parse function is configured in kind.yml, which gives some
flexibility for future implementations of alternative syntaxes.
MozReview-Commit-ID: EQlE6q5E8z7
Considering docker images contents depend very much on the moment they
were built, it is possible that two images with the same hash in the
taskcluster index (at different levels) have different contents. When
this happens, the build or test results could be significantly
different between e.g. try and mozilla-central, possibly leading to
misleading results at landing time.
So if for some reason multiple levels have images for the same hash, the
one used at the highest level should be prefered, such that try uses the
same as mozilla-central once mozilla-central generates the image for
that hash, even if there is an image previously generated for try.
We're adding a dummy servo/ports/geckoservo/ directory, which would make
the filtering logic for Taskcluster tasks think that we want to run
Servo tasks all the time. But we don't have a complete installation of
Servo, so things will inevitably fall over if we did that. To avoid
this situation, change the path that the filter checks for to something
a little more specific and less likely to cause conflicts.
This adds `.cron.yml` and a new mach command to interpret it. While
functionality is limited to nightlies right now, there is room to expand to
more diverse periodic tasks. Let your imagination run wild!
MozReview-Commit-ID: KxQkaUbsjQs
This adds a HASH file next to the VERSION file in the image
context folders for prebuilt docker images. And uses the
HASH for referencing the image in the tasks created by
the decision task.
This way docker will validate the image hash when pulling it
in production. Thus, attackers won't be able to inject code
by compromising the remote docker registries we use to store
prebuilt images. Further more, this makes validation of the
Chain-Of-Trust artifacts easier as this eliminates the need
for whitelists and hash validation.
MozReview-Commit-ID: FD3B9MyeU9Q
This adds an optional "platforms" key to the job description. It can be used in conjunction with
"by-platform" like so:
platforms:
- linux
- windows
worker-type:
by-platform:
linux: ...
windows: ...
worker:
by-platform:
linux: ...
windows: ...
MozReview-Commit-ID: JwL1NAR4bnY
Stylo automation doesn't work unless Servo is present in the source
directory. This commit introduces a "check_servo" filter that prunes
tasks requiring Servo. Currently, this is implemented as a test against
platforms that are unique to Servo.
The use of relative path checking to find the topsrcdir is a bit
unfortunate. But we use this pattern elsewhere in this code.
MozReview-Commit-ID: IRtd53tudJW
Previously, all callers outside of tests that passed
"target_tasks_method" to TaskGraphGenerator all used the same pattern
of looking for a key in the parameters and calling a function in
the target_tasks module.
Future commits will refactor how target tasks graph work. To
make the transition easier, we move the logic for obtaining the
target tasks method into TaskGraphGenerator.
MozReview-Commit-ID: 3QU09iGhoXh
* Compress docker images with zstd
* Removed need for context.tar from decision task
* Index images by level rather than project
MozReview-Commit-ID: 4RL4QXNWmpd
Often we need to setup a test configuration for different platforms, but
not for different platform builds (opt/debug). This leads to cumbersome
configuration duplicates. This patch adds support for platform regular
expression matching. For example, if you want to configure the chunk
size for linux64 platform both for opt and debug, originally you would
do this:
chunks:
by-test-platform:
linux64/opt: 4
linux64/debug: 4
default: 8
With regular expression matching, you only need:
chunks:
by-test-platform:
linux64/.*: 4
default: 8
This patch was originally written by Geoffrey Brown for Windows support.
MozReview-Commit-ID: KbMHV7UkTLe
The `from_parameters` method was never used, and let do confusion over the role
of these parameters. Now there are only two, and they are always required.
MozReview-Commit-ID: AbPqijXucu5
Often we need to setup a test configuration for different platforms, but
not for different platform builds (opt/debug). This leads to cumbersome
configuration duplicates. This patch adds support for platform regular
expression matching. For example, if you want to configure the chunk
size for linux64 platform both for opt and debug, originally you would
do this:
chunks:
by-test-platform:
linux64/opt: 4
linux64/debug: 4
default: 8
With regular expression matching, you only need:
chunks:
by-test-platform:
linux64/.*: 4
default: 8
This patch was originally written by Geoffrey Brown for Windows support.
MozReview-Commit-ID: HFP52N9Ef0k
If you want to run tests on Linux32 and Linux64, you use trychooser syntax like
"-u all[Ubuntu]" which taskcluster supports, but if you also built ASan, and
want tests on ASan, there's no supported documented way to get them. That sounds
like an edge-case, except that to get tests to run on WinXP or Win8 you have to
explicitly list them, so to get tests on every platform you build you have to
list all of them, and if ASan is one of them you have to somehow know to dig
around in the .yml and to know that linux64-asan is the thing to dig out.
MozReview-Commit-ID: 2REf0cUWmK8
