These two new attributes will help us determine which tasks belong in which release promotion graphs. In the future, we'll specify these for all shippable builds as well, and we can reduce the usage of the `product` keys. However, sometimes we need `product` to differ from `shipping-product` (e.g., `mobile` vs `fennec`; in this case we mean `stage_product` vs `shipping_product`), so I haven't yet touched those key/value pairs.
MozReview-Commit-ID: LEuf4CS277Q
In bug 1412690, Dustin noted that the scopes and routes don't belong at the worker level.
To deal with the release indexes, we now have a new `index_builder`. We also add the default
release bbb scope in `build_buildbot_bridge_payload`.
We can and should still move the product info to attributes. I left that for another patch.
MozReview-Commit-ID: 4ZqvnY577S7
Tasks that have the 'always_target' attribute set will be always be included
in the target_task_graph, regardless of target task filtering.
Furthermore, if they were only added because of this attribute (i.e, the
filters would have excluded the task), then the task will be a candidate for
optimization even if the 'optimize_target_tasks' parameter is False.
MozReview-Commit-ID: 9eoVJ5qpAMO
This patch adds the `release-notify-promote` and `release-notify-publish` kinds. It also genericizes all the notifications, and updates the kinds that use those notifications.
MozReview-Commit-ID: 9ymXKzthVF4
This patch adds the beetmover-cdns kind, and adds it to
`publish_fennec`.
This was the first non-buildbot-bridge, non-dummy relpro task, so this
needed a new transform.
This patch also updates the `previous_graph_kinds` and updates the
beetmover scopes in scriptworker.py.
MozReview-Commit-ID: 3rpkjuLjjXz
This patch adds per-task pulse notifications, as well as general support
to handle them.
Longer term we may move away from pulse-notifications, but this allows
us to proceed with pulse-notifications until that time.
MozReview-Commit-ID: 1uB4X682yLT
This patch adds the initial `release-bouncer-aliases` kind, and adds it
to the `publish_fennec` `target_tasks_method`.
It also adds the ability to specify the `tuxedo_server_url`
`by-project`.
MozReview-Commit-ID: 9I4IaUlbCCD
This patch adds the first releasetask as a new kind. To support this, we
added a new `release-promotion` flag in the buildbot job. If this is
set, we use the new `bb_release_worker` function; otherwise we fall back
to the `bb_ci_worker` function (this is the old behavior, factored out
into a separate function).
We also added `build_number` and `release_promotion` attributes in the
task definition. Finally, `build_number` now defaults to 1, allowing us
to create the task graph locally without forcing us to set
`BUILD_NUMBER` in the environment.
MozReview-Commit-ID: 8vNMHJemqAG
We now have a "source" task that produces JSON files with per-file
Bugzilla components and a list of files missing a declared Bugzilla
component. gzip variations are also produced.
The files are published in the index so clients can query e.g.
gecko.v2.mozilla-central.latest.source.source-bugzilla-info/public/components.json
and get the latest metadata. This should help alleviate the need for
querying the moz.build evaluation API on hg.mozilla.org - or at least
facilitate bulk queries of the data from a static source.
MozReview-Commit-ID: 9fAoPSt4bxq
This adds a new `tc-A` Tree Herder group, similar to the `A` Autophone
(or Android) group. (I don't want to include a `g` for Gradle because
eventually there will be nothing that is _not_ Gradle.)
Per https://bugzilla.mozilla.org/show_bug.cgi?id=1371445#c31, the
sheriffs are satisfied with the test output.
As to the rest of
https://bugzilla.mozilla.org/show_bug.cgi?id=1372075#c0, the
documentation is in place, and |mach try fuzzy| has eclipsed
trychooser, so I think we should not update trychooser.
MozReview-Commit-ID: 2OWDEmGCd11
This adds some new optimization strategies. For tests, we use Either(SETA,
SkipUnlessSchedules), thereby giving both mechanisms a chance to skip tasks. On
try, SETA is omitted.
MozReview-Commit-ID: GL4tlwyeBa6
It is not at *all* clear how multiple optimizations for a single task should
interact. No simple logical operation is right in all cases, and in fact in
most imaginable cases the desired behavior turns out to be independent of all
but one of the optimizations. For example, given both `seta` and
`skip-unless-files-changed` optimizations, if SETA says to skip a test, it is
low value and should be skipped regardless of what files have changed. But if
SETA says to run a test, then it has likely been skipped in previous pushes, so
it should be run regardless of what has changed in this push.
This also adds a bit more output about optimization, that may be useful for
anyone wondering why a particular job didn't run.
MozReview-Commit-ID: 3OsvRnWjai4
This adds some new optimization strategies. For tests, we use Either(SETA,
SkipUnlessSchedules), thereby giving both mechanisms a chance to skip tasks. On
try, SETA is omitted.
MozReview-Commit-ID: GL4tlwyeBa6
It is not at *all* clear how multiple optimizations for a single task should
interact. No simple logical operation is right in all cases, and in fact in
most imaginable cases the desired behavior turns out to be independent of all
but one of the optimizations. For example, given both `seta` and
`skip-unless-files-changed` optimizations, if SETA says to skip a test, it is
low value and should be skipped regardless of what files have changed. But if
SETA says to run a test, then it has likely been skipped in previous pushes, so
it should be run regardless of what has changed in this push.
This also adds a bit more output about optimization, that may be useful for
anyone wondering why a particular job didn't run.
MozReview-Commit-ID: 3OsvRnWjai4
We currently vary the cache name for run-task tasks whenever run-task
changes. This allows us to not worry about backwards or forwards
compatibility of caches in run-task tasks.
This strategy doesn't work for out-of-tree Docker images because
the content of run-task cannot be determined at Taskgraph time:
the content of run-task was determined when that Docker image was
built and there is no way to get that content efficiently during
Taskgraph.
So, for out-of-tree Docker images we now vary the cache name by
the Docker image value, which includes its name and a tag or
hash. This means that out-of-tree run-task tasks will get separate
caches for each distinct Docker image.
This isn't ideal. Ideally we would share caches if run-task doesn't
vary between Docker images. But without any way of proving that
at Taskgraph time, we take the safe road and force cache separation.
MozReview-Commit-ID: FMiQBqfvjqW
