This decreases the scheduling algorithm from a CT_MEDIUM (0.8) threshold to
CT_LOW (0.7). It also reduces the threshold used in the manifest resolving
algorithm down to 0.5. This means we'll have more "ride-along" manifests that
happen to be in the same chunk as the more important ones.
Differential Revision: https://phabricator.services.mozilla.com/D94757
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Sets autoland to use the 'bugbug' test manifest loader. This is being enabled
as part of a temporary trial to see the impact it has on sheriffing.
Differential Revision: https://phabricator.services.mozilla.com/D90160
In order for Thunderbird to effectively cut down on excess builds, there needs
to be a way to define taskgraph optimization strategies beyond what is made
available in the mozilla repository.
taskgraph.optimize.register_strategy gets half of the job done. The other piece
is task schema validation which checks optimizations against OptimizationSchema.
OptimizationSchema gets moved to taskgraph.optimize.schema so that it is not set
too early by taskgraph.util.schema. Projects then call set_optimization_schema()
to replace the default schema object. set_optimization_schema() needs to be called
in the project's taskgraph:register function before any transform code is loaded.
Differential Revision: https://phabricator.services.mozilla.com/D88429
Sets autoland to use the 'bugbug' test manifest loader. This is being enabled
as part of a temporary trial to see the impact it has on sheriffing.
Depends on D90159
Differential Revision: https://phabricator.services.mozilla.com/D90160
In the past, the 'backstop' optimization was applied to tasks by default across
all projects, even though it only really made sense on autoland. To choose what
would happen on non-autoland branches, we invented this 'remove_on_projects'
concept.
These days, we only apply the backstop optimization in the first place for
autoland. So 'remove_on_projects' is no longer necessary.
Depends on D88149
Differential Revision: https://phabricator.services.mozilla.com/D88150
This removes the last uses of the 'push-interval-10' and 'push-interval-20' strategies.
They are being removed because they are dangerous in that its easy to accidentally not run
tasks when they should.
Instead, task authors should decide whether they want their tasks to run on
"backstop" pushes (run everything) or "expanded" pushes (run more than usual,
but still not as much as a backstop). Note that using "expanded" means the task
will *also* run on backstop pushes. It'll just additionally run on "expanded"
pushes.
In practice 'backstop' pushes will be every 20th push and 'expanded' pushes
will be every 10th push. Though this may vary due to the time component in
backstops.
Differential Revision: https://phabricator.services.mozilla.com/D89503
This is a nomenclature change + refactoring. Now there is only a single
"backstop" push. Which is currently set to every 20th push on autoland (or
every push on non-autoland branches).
Now there is also a concept of an "expanded" push. These are pushes that run
more stuff than usual, but not as much as a backstop normally would. These are
currently set to run at half the interval of a backstop.
Concretely, here are the strategy changes:
* Renamed 'backstop' -> 'skip-unless-backstop'
* Introduced the 'skip-unless-expanded' strategy which has 'skip-unless-backstop' baked in
* Merged 'optimized-backstop' into the 'test' strategy (with expanded comments)
* Baked both 'skip-unless-expanded' and 'skip-unless-backstop' into the 'test' strategy
Differential Revision: https://phabricator.services.mozilla.com/D89501
It turns out that 'Not' is needed to negate "backstops". E.g, we normally
we want to use a pattern like so:
All("skip-unless-backstop", "test")
Since 'skip-unless-backstop' returns False on backstop pushes, it disables
the test strategy there.
However, suppose we wanted to run a special optimization, *only* on backstop
pushes. I.e, the opposite of the above example. Then we need to use:
All(Not("skip-unless-backstop"), "test-backstop")
Depends on D89500
Differential Revision: https://phabricator.services.mozilla.com/D89734
We want to try to align 'push-interval' tasks to the 'backstop'. This way
we have greater confidence in our backstop pushes, and it will allow us to
simplify a lot of our backstop logic.
Depends on D89055
Differential Revision: https://phabricator.services.mozilla.com/D89056
We want to be able to retroactively tell whether a push was a backstop or not.
This patch stores whether or not a push was a "backstop" directly in the
parameters. The optimization strategy now simply returns 'not
params["backstop"]'.
For simplicity, I'm not counting the 'optimized-backstop' as a backstop. It's
unclear if we'll want to be able to detect these types of the pushes in the
future or not, but we can cross that bridge when we get there.
Differential Revision: https://phabricator.services.mozilla.com/D88151
In the past, the 'backstop' optimization was applied to tasks by default across
all projects, even though it only really made sense on autoland. To choose what
would happen on non-autoland branches, we invented this 'remove_on_projects'
concept.
These days, we only apply the backstop optimization in the first place for
autoland. So 'remove_on_projects' is no longer necessary.
Depends on D88149
Differential Revision: https://phabricator.services.mozilla.com/D88150
This patch cleans up some of the backstop strategy names. Specifically:
1. Rename 'full-backstop' -> 'backstop'. The old 'backstop' algorithm was
unused anyway, so there is no conflict. It is also just defined directly in
the decorator rather than using 'Alias'.
So now rather than 'full-backstop' and 'optimized-backstop', it's just
'backstop' and 'optimized-backstop'.
2. Remove 'backstop-X-hours-Y-minutes' strategies, and replace them with
the corresponding 'push-interval-X' strategy.
This means we lose the time component in the 'optimized-backstop'. But it isn't
a problem, because we shouldn't be using a time component there at all anyway
(we should just use it with the 'backstop').
Differential Revision: https://phabricator.services.mozilla.com/D88149
Certain tasks are meant to perform follow-up work to another task. For example,
a 'signing' task signs an artifact from a dependency task. An 'upload-symbols'
tasks uploads an artifact from a dependency task.
In general, we only want to run these types of tasks when the dependency task
is running. But in practice, these tasks often cause the dependency to get
pulled in.
We've tried to fix this in the past by making the 'run-on-projects' and
'optimization' keys match their primary dependency. But it's very easy to
mess this up, and some optimizations (like bugbug) take the content of the
task definition into account. So even if both tasks use the same optimization,
they could have different results.
This revision adds a new 'if-dependencies' key that denotes a task that should
only run if one of the dependencies specified in this list is also run. It
will allow us to stop playing whack-a-mole when trying to make sure tasks
aren't being pulled in solely due to these types of dependencies.
This feature is implemented independently of the 'optimization', so tasks that
specify this can still be optimized even if their primary dependency does run.
Differential Revision: https://phabricator.services.mozilla.com/D80508
This is needed as future revisions will refactor the optimization process to be
two pass. So we can't be sure if a task is optimized or not until the very end.
Depends on D87771
Differential Revision: https://phabricator.services.mozilla.com/D87772
Sets autoland to use the 'bugbug' test manifest loader. This is being enabled
as part of a temporary trial to see the impact it has on sheriffing.
Differential Revision: https://phabricator.services.mozilla.com/D85011
Sets autoland to use the 'bugbug' test manifest loader. This is being enabled
as part of a temporary trial to see the impact it has on sheriffing.
Differential Revision: https://phabricator.services.mozilla.com/D85011
This creates a new set of optimization strategies
(taskgraph.optimize:project.autoland) to use with autoland. Among other things,
it also means there's no need for the 'test-try' optimization as the autoland
strategies are no longer the default behaviour.
Differential Revision: https://phabricator.services.mozilla.com/D79705
