Commit Graph

212 Commits

Author SHA1 Message Date
Tom Ritter
346da0ff52 Bug 1674182 - Support a regex filter for./mach try auto r=ahal
Differential Revision: https://phabricator.services.mozilla.com/D95186
2020-11-05 17:53:31 +00:00
Ricky Stewart
31b8ca9716 Bug 1654103: Standardize on Black for Python code in mozilla-central.
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.

To produce this patch I did all of the following:

1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.

2. Run ./mach lint --linter black --fix

3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.

4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.

5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).

# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D94045
2020-10-26 18:34:53 +00:00
Bogdan Tara
612312a64c Backed out 10 changesets (bug 1654103, bug 1672023, bug 1518999) for PanZoomControllerTest.touchEventForResult gv-junit failures CLOSED TREE
Backed out changeset ff3fb0b4a512 (bug 1672023)
Backed out changeset e7834b600201 (bug 1654103)
Backed out changeset 807893ca8069 (bug 1518999)
Backed out changeset 13e6b92440e9 (bug 1518999)
Backed out changeset 8b2ac5a6c98a (bug 1518999)
Backed out changeset 575748295752 (bug 1518999)
Backed out changeset 65f07ce7b39b (bug 1518999)
Backed out changeset 4bb80556158d (bug 1518999)
Backed out changeset 8ac8461d7bd7 (bug 1518999)
Backed out changeset e8ba13ee17f5 (bug 1518999)
2020-10-24 03:36:18 +03:00
Ricky Stewart
065d2eb893 Bug 1654103: Standardize on Black for Python code in mozilla-central. r=remote-protocol-reviewers,marionette-reviewers,webdriver-reviewers,perftest-reviewers,devtools-backward-compat-reviewers,jgilbert,preferences-reviewers,sylvestre,maja_zf,webcompat-reviewers,denschub,ntim,whimboo,sparky
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.

To produce this patch I did all of the following:

1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.

2. Run ./mach lint --linter black --fix

3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.

4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.

5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).

# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D94045
2020-10-23 20:40:42 +00:00
Dorel Luca
95b44c982f Backed out changeset 7558c8821a07 (bug 1654103) for multiple failures. CLOSED TREE 2020-10-22 03:51:06 +03:00
Ricky Stewart
43baed3c18 Bug 1654103: Standardize on Black for Python code in mozilla-central. r=remote-protocol-reviewers,marionette-reviewers,webdriver-reviewers,perftest-reviewers,devtools-backward-compat-reviewers,jgilbert,preferences-reviewers,sylvestre,maja_zf,webcompat-reviewers,denschub,ntim,whimboo,sparky
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.

To produce this patch I did all of the following:

1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.

2. Run ./mach lint --linter black --fix

3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.

4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).

# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D94045
2020-10-21 21:27:27 +00:00
Andrew Halberstadt
16b601cc2a Bug 1643689 - [taskgraph] enable manifest-scheduling on autoland, r=marco
Sets autoland to use the 'bugbug' test manifest loader. This is being enabled
as part of a temporary trial to see the impact it has on sheriffing.

Differential Revision: https://phabricator.services.mozilla.com/D90160
2020-09-30 13:27:14 +00:00
Andrew Halberstadt
7c01f31a5b Bug 1666809 - Insert decision task indexes directly via taskluster API rather than index-task, r=taskgraph-reviewers,aki
We started using the "backstop" index added by bug 1660506 to determine whether
a push should be a backstop based on a time interval. The problem is that this
index gets added by an index-task that runs after the decision task has
completed.  Therefore, if two pushes land at roughly the same time (i.e, the
second decision task starts before the first completes), then they can both
determine themselves as backstops.

This patch gets around the problem by inserting the "backstop" index as early
as possible (immediately after resolving parameters), so the chances of this
happening become very low. It's still theoretically possible that it could
happen again, but we don't need this to be 100% perfect. As long as it is rare,
it's good enough.

Depends on D91191

Differential Revision: https://phabricator.services.mozilla.com/D91192
2020-09-24 19:19:36 +00:00
Mihai Alexandru Michis
7880c3527d Backed out changeset 0b196026ed59 (bug 1643689) for causing issues with manifest scheduling.
CLOSED TREE
2020-09-17 11:26:12 +03:00
Andrew Halberstadt
4f67f40e40 Bug 1643689 - [taskgraph] enable manifest-scheduling on autoland, r=marco
Sets autoland to use the 'bugbug' test manifest loader. This is being enabled
as part of a temporary trial to see the impact it has on sheriffing.

Depends on D90159

Differential Revision: https://phabricator.services.mozilla.com/D90160
2020-09-14 23:12:31 +00:00
Mihai Tabara
26ab1edc56 Bug 1661673 - remove esr68 references.r=releng-reviewers,aki DONTBUILD
Differential Revision: https://phabricator.services.mozilla.com/D88618
2020-08-31 07:59:28 +00:00
Andrew Halberstadt
1a5d28b93c Bug 1660506 - Move backstop determination to the decision task and store it in a parameter, r=marco,taskgraph-reviewers,jmaher
We want to be able to retroactively tell whether a push was a backstop or not.
This patch stores whether or not a push was a "backstop" directly in the
parameters. The optimization strategy now simply returns 'not
params["backstop"]'.

For simplicity, I'm not counting the 'optimized-backstop' as a backstop. It's
unclear if we'll want to be able to detect these types of the pushes in the
future or not, but we can cross that bridge when we get there.

Differential Revision: https://phabricator.services.mozilla.com/D88151
2020-08-26 21:56:05 +00:00
Nick Thomas
7e639781ec Bug 1630809 - add initial support for partner attribution in release promotion r=aki
Renames the release_enable_partners parameter to release_enable_partner_repack, and adds release_enable_partner_attribution for attribution. This it to provide support for disabling them independently in main releases, and in respins.
Adds docs for attribution, update docs for repacks.
Hardwire values for the enable params for the respin flavors, other wise read from the input (defaulting to on in promotion, off otherwise).
Fixes up the rebuild-kinds for partner repacks so that they reflect the current set, although the top level may be all that is needed.

Differential Revision: https://phabricator.services.mozilla.com/D87727
2020-08-27 22:19:41 +00:00
Butkovits Atila
797523cdf6 Backed out 9 changesets (bug 1630809, bug 1653476) for Gecko Decision failures. CLOSED TREE
Backed out changeset 02a27bfc76dd (bug 1653476)
Backed out changeset afb5df61943a (bug 1630809)
Backed out changeset 04628c1f98e9 (bug 1630809)
Backed out changeset 4b4d50e0b1bf (bug 1630809)
Backed out changeset 2fa2deb5c993 (bug 1630809)
Backed out changeset d6652114cac3 (bug 1630809)
Backed out changeset ad5e4caa3291 (bug 1630809)
Backed out changeset d3d841cd14f3 (bug 1630809)
Backed out changeset b3746502e227 (bug 1630809)
2020-08-28 01:15:03 +03:00
Nick Thomas
dccead459d Bug 1630809 - add initial support for partner attribution in release promotion r=aki
Renames the release_enable_partners parameter to release_enable_partner_repack, and adds release_enable_partner_attribution for attribution. This it to provide support for disabling them independently in main releases, and in respins.
Adds docs for attribution, update docs for repacks.
Hardwire values for the enable params for the respin flavors, other wise read from the input (defaulting to on in promotion, off otherwise).
Fixes up the rebuild-kinds for partner repacks so that they reflect the current set, although the top level may be all that is needed.

Differential Revision: https://phabricator.services.mozilla.com/D87727
2020-08-27 05:28:53 +00:00
Razvan Maries
2a46b35060 Backed out changeset 9251996ce8c9 (bug 1657939) for Gecko Decision Task failures. CLOSED TREE 2020-08-11 00:32:44 +03:00
Justin Wood
9f41cda459 Bug 1657939 - Output an artifact that identifies replaced taskID's. r=ahal
Differential Revision: https://phabricator.services.mozilla.com/D86379
2020-08-10 19:20:28 +00:00
Joel Maher
d378cfab4e Bug 1643689 - disable 1st round of manifest scheduling. r=aryx
disable 1st round of manifest scheduling

Differential Revision: https://phabricator.services.mozilla.com/D85621
2020-07-31 20:15:34 +00:00
Andrew Halberstadt
fe90b95e98 Bug 1643689 - [taskgraph] enable manifest-scheduling on autoland, r=marco
Sets autoland to use the 'bugbug' test manifest loader. This is being enabled
as part of a temporary trial to see the impact it has on sheriffing.

Differential Revision: https://phabricator.services.mozilla.com/D85011
2020-07-29 17:05:48 +00:00
Bogdan Tara
20b0b6082e Backed out changeset 9be5f086895c (bug 1643689) for busting gecko decision task and causig bug 1655807 CLOSED TREE 2020-07-28 19:46:47 +03:00
Andrew Halberstadt
81eb3647a9 Bug 1643689 - [taskgraph] enable manifest-scheduling on autoland, r=marco
Sets autoland to use the 'bugbug' test manifest loader. This is being enabled
as part of a temporary trial to see the impact it has on sheriffing.

Differential Revision: https://phabricator.services.mozilla.com/D85011
2020-07-28 14:40:40 +00:00
Andrew Halberstadt
c27e14c6dd Bug 1636271 - Fix flake8 issue arising from merge conflict CLOSED TREE
Differential Revision: https://phabricator.services.mozilla.com/D85155
2020-07-28 14:15:27 +00:00
Andrew Halberstadt
4a8bbd96ce Bug 1636271 - [taskgraph] Set autoland optimizations via per-project parameter, r=tomprince
This creates a new set of optimization strategies
(taskgraph.optimize:project.autoland) to use with autoland. Among other things,
it also means there's no need for the 'test-try' optimization as the autoland
strategies are no longer the default behaviour.

Differential Revision: https://phabricator.services.mozilla.com/D79705
2020-07-28 13:22:48 +00:00
Andrew Halberstadt
b462d16eef Bug 1636271 - [taskgraph] Move 'optimize-strategies' from try_task_config.json to a parameter, r=tomprince
Differential Revision: https://phabricator.services.mozilla.com/D79704
2020-07-28 13:21:05 +00:00
Ben Hearsum
a517f66228 Bug 1650208: run opt builds by default on autoland; shippable builds by default on central r=tomprince,ahal,marco
Differential Revision: https://phabricator.services.mozilla.com/D83186
2020-07-27 16:25:40 +00:00
Narcis Beleuzu
f6c2e1b1c1 Backed out 3 changesets (bug 1636271) for breaking the Gecko Decision Task. CLOSED TREE
Backed out changeset 4b0f13fcf941 (bug 1636271)
Backed out changeset 4250f49877ba (bug 1636271)
Backed out changeset d3f93bd6b1f9 (bug 1636271)
2020-07-22 23:40:32 +03:00
Andrew Halberstadt
bee55a2da6 Bug 1636271 - [taskgraph] Set autoland optimizations via per-project parameter, r=tomprince
This creates a new set of optimization strategies
(taskgraph.optimize:project.autoland) to use with autoland. Among other things,
it also means there's no need for the 'test-try' optimization as the autoland
strategies are no longer the default behaviour.

Differential Revision: https://phabricator.services.mozilla.com/D79705
2020-07-22 17:09:02 +00:00
Andrew Halberstadt
7d9e380699 Bug 1636271 - [taskgraph] Move 'optimize-strategies' from try_task_config.json to a parameter, r=tomprince
Differential Revision: https://phabricator.services.mozilla.com/D79704
2020-07-22 15:36:55 +00:00
Tom Prince
938a7e6d6c Bug 1638990: [taskgraph] Apply appropriate change from python-modernize; r=ahal
Differential Revision: https://phabricator.services.mozilla.com/D84086
2020-07-20 19:27:46 +00:00
Tom Prince
88fbb85a23 Bug 1638990: [taskgraph] Make taskgraph.decision.write_artifact work on python 3; r=ahal
Differential Revision: https://phabricator.services.mozilla.com/D84083
2020-07-20 19:27:34 +00:00
Tom Prince
8d86a91648 Bug 1638990: [taskgraph] Make test_util_runnable_jobs pass on python 3; r=ahal
Differential Revision: https://phabricator.services.mozilla.com/D84077
2020-07-20 19:27:09 +00:00
Tom Prince
393924276c Bug 1626058: Add support for writting out artifacts from transforms in the decision task; r=Callek,ahal
To support using kaniko[1] for building images, we need to generate the docker
contexts in a seprate task from the docker-image task. Since we use the hash of
the context as the cached-task digest, we generate the context in the decision
task already, so this adds a way to write that out to be used by downstream
tasks.

[1] https://github.com/GoogleContainerTools/kaniko

Differential Revision: https://phabricator.services.mozilla.com/D77839
2020-07-07 19:56:11 +00:00
Rob Lemley
853da54663 Bug 1646843 - Add comm-esr78 support to taskgraph. r=Callek
Follow-up to bug 1646588 to support comm-esr78 repository.

Differential Revision: https://phabricator.services.mozilla.com/D80271
2020-06-18 20:42:03 +00:00
Justin Wood
050683fd9d Bug 1646588 - Add esr78 support. r=aki
Differential Revision: https://phabricator.services.mozilla.com/D79731
2020-06-18 19:57:20 +00:00
Tom Prince
d535cab6de Bug 1641971: [taskgraph] Always pass an explicit decision task id to create_tasks; r=Callek
Differential Revision: https://phabricator.services.mozilla.com/D77551
2020-06-03 02:35:03 +00:00
Tom Prince
d69a6151ef Bug 1641971: [taskgraph] Pass explicit decision task id to action.json generation; r=Callek
Differential Revision: https://phabricator.services.mozilla.com/D77550
2020-06-03 02:35:05 +00:00
Tom Prince
84b805501b Bug 1641971: Pass mach perftest options explicitly, rather than inpsect parameters; r=tarek
Differential Revision: https://phabricator.services.mozilla.com/D77548
2020-06-03 02:35:09 +00:00
Tom Prince
d648744c0d Bug 1641971: Allow references to decision task via task-reference/artifact-reference; r=Callek
Differential Revision: https://phabricator.services.mozilla.com/D77547
2020-06-03 02:43:34 +00:00
Narcis Beleuzu
982ec4412a Backed out 7 changesets (bug 1641971) for breaking Gecko Decision Task opt . CLOSED TREE
Backed out changeset a039e60a26d2 (bug 1641971)
Backed out changeset e7d4c490c31a (bug 1641971)
Backed out changeset f2322981b294 (bug 1641971)
Backed out changeset d123b08d2c20 (bug 1641971)
Backed out changeset 57772c0b9b3e (bug 1641971)
Backed out changeset 11a887f73dbb (bug 1641971)
Backed out changeset e9c7a59d4bc5 (bug 1641971)
2020-06-03 05:30:51 +03:00
Tom Prince
b4ec09d781 Bug 1641971: [taskgraph] Always pass an explicit decision task id to create_tasks; r=Callek
Differential Revision: https://phabricator.services.mozilla.com/D77551
2020-06-02 18:23:19 +00:00
Tom Prince
4b27ad1785 Bug 1641971: [taskgraph] Pass explicit decision task id to action.json generation; r=Callek
Differential Revision: https://phabricator.services.mozilla.com/D77550
2020-06-02 18:23:20 +00:00
Tom Prince
bcc9899eab Bug 1641971: Pass mach perftest options explicitly, rather than inpsect parameters; r=tarek
Differential Revision: https://phabricator.services.mozilla.com/D77548
2020-06-02 18:23:07 +00:00
Tom Prince
d9be0f48e6 Bug 1641971: Allow references to decision task via task-reference/artifact-reference; r=Callek
Differential Revision: https://phabricator.services.mozilla.com/D77547
2020-06-02 18:50:10 +00:00
Andrew Halberstadt
297ec4fd75 Bug 1633866 - [taskgraph] Create a system to choose how test manifests are loaded via a parameter, r=tomprince
Currently test manifests are loaded by instantiating a TestResolver and
traversing moz.build files to find manifests.

With 'manifest-scheduling', we'll want to grab the manifests directly from the
bugbug service instead. Initially we'll want to enable manifest-scheduling with
|mach try auto|, but not on autoland yet.

This patch will allow |mach try auto| to set the parameter that causes bugbug
to be used (see future commits in this bug).

Differential Revision: https://phabricator.services.mozilla.com/D76522
2020-05-29 18:51:14 +00:00
Andrew Halberstadt
395c323eb6 Bug 1641283 - [taskgraph] Upload bugbug push schedules API data to an artifact rather than the log, r=marco
Differential Revision: https://phabricator.services.mozilla.com/D77140
2020-05-29 14:39:38 +00:00
Tom Prince
d3f81a7b95 Bug 1628832: [KaiOS] Add kaios specific taskgraph paramters and target tasks; r=Callek
Differential Revision: https://phabricator.services.mozilla.com/D75243
2020-05-14 18:48:00 +00:00
James Graham
84da8eabc8 Bug 1624280 - Add support for routes in try_task_config.json, r=ahal
This adds an optional routes key to the task_task_config schema,
 which is a list of strings. Anything in this key is added to the list
of routes for tasks scheduled by the decision task.

Differential Revision: https://phabricator.services.mozilla.com/D67827
2020-04-01 09:18:37 +00:00
Tom Prince
48be49988a No bug: [taskgraph] Extract common setting from try_options to try_task_config in parameters; r=ahal
There are a number of settings that have equivalent expressions in
`try_options` (used for try syntax) and `try_task_config` (used for other try
selectors). Rather than requiring task generation code to understand both
formats, this converts the try syntax specification to `try_task_config` at
parameter generation time.

Differential Revision: https://phabricator.services.mozilla.com/D66536
2020-03-14 01:18:34 +00:00
Andrew Halberstadt
d2436df34a Bug 1603463 - [tryselect] Add the ability to run 'mach try auto' with alternative scheduling algorithms, r=tomprince
This allows us to change the default optimization strategy used in try pushes.
While probably not super useful to developers, it can help us easily test
changes to new and experimental optimizations on try.

This also changes the default to the 'bugbug_push_schedules' strategy, since
SETA is more or less random and shouldn't be used by 'mach try auto'. In the
future, we'll switch this back to simply using the default optimization as the
default will ideally be the best one that we have.

Differential Revision: https://phabricator.services.mozilla.com/D65746
2020-03-13 20:32:14 +00:00
Andrew Halberstadt
b44c13dac3 Bug 1615333 - [taskgraph] Upload an artifact from the decision task mapping tests to manifests r=tomprince
Differential Revision: https://phabricator.services.mozilla.com/D62822
2020-03-13 20:30:09 +00:00