Commit Graph

139 Commits

Author SHA1 Message Date
Andrew Halberstadt
d91cfd7d38 Bug 1725404 - Refactor '--target-kind' / '--output-file' logic to use standalone taskgraph's implementation, r=bhearsum
This syncs in the latest 'main.py' from standalone taskgraph and refactors
mach_commands.py to use it.

Differential Revision: https://phabricator.services.mozilla.com/D122778
2021-08-17 13:35:12 +00:00
Andrew Halberstadt
c6d39c2686 Bug 1725404 - [taskgraph] Implement 'mach taskcluster-image-digest' command, r=bhearsum
I'm not sure if this is needed in Gecko, but it matches the similar command in
standalone taskgraph so makes the two more in sync. I don't think having it
will hurt.

Differential Revision: https://phabricator.services.mozilla.com/D122527
2021-08-16 14:56:27 +00:00
Andrew Halberstadt
b8cd58c34f Bug 1725404 - [taskgraph] Re-use 'action-callback' subcommand args from main.py, r=bhearsum
Differential Revision: https://phabricator.services.mozilla.com/D122526
2021-08-16 14:56:26 +00:00
Andrew Halberstadt
cdfce8a9b7 Bug 1725404 - [taskgraph] Re-use 'decision' args from main.py, r=bhearsum
Differential Revision: https://phabricator.services.mozilla.com/D122524
2021-08-16 14:56:26 +00:00
Andrew Halberstadt
c7757aecb4 Bug 1725404 - [taskgraph] Use copy of 'get_filtered_taskgraph' from main.py, r=bhearsum
These two functions were identical.

Differential Revision: https://phabricator.services.mozilla.com/D122523
2021-08-16 14:56:26 +00:00
Andrew Halberstadt
becc3978d1 Bug 1725404 - [taskgraph] Re-use 'docker' subcommand args from main.py, r=bhearsum
Differential Revision: https://phabricator.services.mozilla.com/D122522
2021-08-16 14:56:25 +00:00
Andrew Halberstadt
48e9c59a24 Bug 1725404 - [taskgraph] Re-use 'show' subcommand args from main.py, r=bhearsum
Differential Revision: https://phabricator.services.mozilla.com/D122521
2021-08-16 14:56:25 +00:00
Alex Lopez
25a0a17cae Bug 1696251 - Replace self with command_context where possible in existing mach commands. r=mhentges,webdriver-reviewers,perftest-reviewers,whimboo
This step removes all the dependencies of mach commands to
having a MachCommandBase as the `self` by using the `command_context`
argument instead. This also removes any remaining statefulness from those
classes that implement mach commands, ultimately making it easier to move
existing commands out of classes in a follow-up.

Differential Revision: https://phabricator.services.mozilla.com/D118058
2021-07-19 16:04:25 +00:00
Butkovits Atila
44ca1b9737 Backed out changeset e1921c5112d8 (bug 1696251) for causing bustages complaining about 'CommandContext'. CLOSED TREE 2021-07-16 20:35:55 +03:00
Alex Lopez
eaf2553782 Bug 1696251 - Replace self with command_context where possible in existing mach commands. r=mhentges,webdriver-reviewers,perftest-reviewers,whimboo
This step removes all the dependencies of mach commands to
having a MachCommandBase as the `self` by using the `command_context`
argument instead. This also removes any remaining statefulness from those
classes that implement mach commands, ultimately making it easier to move
existing commands out of classes in a follow-up.

Differential Revision: https://phabricator.services.mozilla.com/D118058
2021-07-16 15:51:29 +00:00
Ben Hearsum
d5d998273e No bug: Add helpful suggestion to ./mach taskgraph --diff for some circumstances r=ahal
This helps hint that in some cases, a user may have forgotten to diff
with `-J` if they were changing task bodies/payloads instead of adding
or removing a task.

Differential Revision: https://phabricator.services.mozilla.com/D116384
2021-06-01 14:10:07 +00:00
Rob Lemley
b63366c1ea Bug 1696251 - Fix "mach taskgraph full" command. r=mhentges
Differential Revision: https://phabricator.services.mozilla.com/D115543
2021-05-19 20:50:05 +00:00
Alex Lopez
bc8dd276ea Bug 1696251 - Pass MachCommandBase object as first argument for Mach Commands. r=mhentges,remote-protocol-reviewers,marionette-reviewers,webdriver-reviewers,perftest-reviewers
As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).

Differential Revision: https://phabricator.services.mozilla.com/D109650
2021-05-17 16:15:58 +00:00
Andrew Halberstadt
3120ed6b0b No Bug - [taskgraph] Add 'check=True' to subprocess when running with --diff, r=taskgraph-reviewers,jmaher
Differential Revision: https://phabricator.services.mozilla.com/D113860
2021-05-03 20:55:19 +00:00
Andrew Halberstadt
cd9f0c33f6 No Bug - [taskgraph] Generate current taskgraph before base when running with --diff, r=taskgraph-reviewers,aki
This ensures that when your patch has an error, taskgraph generation will fail
faster rather than needing to wait for the base generation to complete.

Depends on D113854

Differential Revision: https://phabricator.services.mozilla.com/D113859
2021-05-03 20:55:18 +00:00
Andrew Halberstadt
7e90fb3b15 Bug 1702819 - [taskgraph] Support Python 3.6 with ./mach taskgraph <sub> --diff, r=taskgraph-reviewers,jmaher
Differential Revision: https://phabricator.services.mozilla.com/D113854
2021-05-03 20:55:18 +00:00
Butkovits Atila
9cdb8e2e2a Backed out 2 changesets (bug 1696251) for causing js-bench-sm failures. CLOSED TREE
Backed out changeset 1c84c9a34575 (bug 1696251)
Backed out changeset e169193b7423 (bug 1696251)
2021-04-23 02:53:36 +03:00
Alex Lopez
cb583c215f Bug 1696251 - Pass MachCommandBase object as first argument for Mach Commands. r=mhentges,remote-protocol-reviewers,marionette-reviewers,webdriver-reviewers,perftest-reviewers
As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).

Differential Revision: https://phabricator.services.mozilla.com/D109650
2021-04-22 18:56:15 +00:00
Cosmin Sabou
9c545e95b4 Backed out changeset d4a5d8567977 (bug 1696251) for non-unified build bustages. CLOSED TREE 2021-04-19 19:43:31 +03:00
Alex Lopez
4eb9b0babc Bug 1696251 - Pass MachCommandBase object as first argument for Mach Commands. r=mhentges,remote-protocol-reviewers,marionette-reviewers,webdriver-reviewers,perftest-reviewers
As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).

Differential Revision: https://phabricator.services.mozilla.com/D109650
2021-04-19 16:15:11 +00:00
Ben Hearsum
02bd2587b7 No bug: ensure ./mach taskgraph --diff returns you to your starting branch when using git. r=ahal
Differential Revision: https://phabricator.services.mozilla.com/D111848
2021-04-14 13:49:01 +00:00
Andrew Halberstadt
0f9578d90b Bug 1409733 - [taskcluster] Add a mach setting to override the diff command in |mach taskgraph --diff|, r=taskgraph-reviewers,bhearsum
Differential Revision: https://phabricator.services.mozilla.com/D107542
2021-03-09 16:18:25 +00:00
Andrew Halberstadt
da850de6a8 Bug 1409733 - [taskcluster] Implement a --diff flag on |mach taskgraph|, r=taskgraph-reviewers,bhearsum
This implements a --diff flag on the suite of |mach taskgraph| commands. E.g:

    ./mach taskgraph target -p project=autoland -J --diff

The above will update to the base revision of your stack (e.g the public
revision), generate the taskgraph, update to the current revision, generate the
taskgraph again, then display a diff of the two.

You can also specify a specifier to arg, e.g:

    ./mach taskgraph target -J --diff .~1

This will diff against the parent revision in mercurial (use 'HEAD~1' for git).

Differential Revision: https://phabricator.services.mozilla.com/D107274
2021-03-09 16:18:24 +00:00
Andrew Halberstadt
56a33667c6 Bug 1409733 - [taskcluster] Refactor |mach taskgraph| generation into separate format function, r=taskgraph-reviewers,bhearsum
This will allow us to more easily generate and format the taskgraph multiple
times from the same session (e.g, needed for diffing multiple graphs).

Differential Revision: https://phabricator.services.mozilla.com/D107273
2021-03-09 16:18:24 +00:00
Andrew Halberstadt
96c042070b Bug 1409733 - [taskcluster] Sort imports in |mach_commands.py|, r=taskgraph-reviewers,bhearsum
This was generated using isort and pyright.

Differential Revision: https://phabricator.services.mozilla.com/D107272
2021-03-09 16:18:24 +00:00
Mitchell Hentges
94604ead81 Bug 1661153: Removes _ensure_zstd() and usages r=firefox-build-system-reviewers,glandium
Removes a bunch of zstandard installations that were ran from
within the mach virtualenv, which should already have zstandard
installed.

Differential Revision: https://phabricator.services.mozilla.com/D98387
2020-12-03 22:58:52 +00:00
Cosmin Sabou
73a291a894 Backed out changeset f1d785de7ab3 (bug 1661153) for causing symbols bustages. a=backout. 2020-12-03 00:49:17 +02:00
Mitchell Hentges
bf9c59931b Bug 1661153: Removes _ensure_zstd() and usages r=firefox-build-system-reviewers,glandium
The zstandard package is always installed in the mach virtualenv.
The patch assumes that zstandard is only used from the mach virtualenv,
and never the build virtualenv.

Differential Revision: https://phabricator.services.mozilla.com/D98387
2020-12-02 15:16:14 +00:00
Ricky Stewart
31b8ca9716 Bug 1654103: Standardize on Black for Python code in mozilla-central.
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.

To produce this patch I did all of the following:

1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.

2. Run ./mach lint --linter black --fix

3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.

4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.

5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).

# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D94045
2020-10-26 18:34:53 +00:00
Bogdan Tara
612312a64c Backed out 10 changesets (bug 1654103, bug 1672023, bug 1518999) for PanZoomControllerTest.touchEventForResult gv-junit failures CLOSED TREE
Backed out changeset ff3fb0b4a512 (bug 1672023)
Backed out changeset e7834b600201 (bug 1654103)
Backed out changeset 807893ca8069 (bug 1518999)
Backed out changeset 13e6b92440e9 (bug 1518999)
Backed out changeset 8b2ac5a6c98a (bug 1518999)
Backed out changeset 575748295752 (bug 1518999)
Backed out changeset 65f07ce7b39b (bug 1518999)
Backed out changeset 4bb80556158d (bug 1518999)
Backed out changeset 8ac8461d7bd7 (bug 1518999)
Backed out changeset e8ba13ee17f5 (bug 1518999)
2020-10-24 03:36:18 +03:00
Ricky Stewart
065d2eb893 Bug 1654103: Standardize on Black for Python code in mozilla-central. r=remote-protocol-reviewers,marionette-reviewers,webdriver-reviewers,perftest-reviewers,devtools-backward-compat-reviewers,jgilbert,preferences-reviewers,sylvestre,maja_zf,webcompat-reviewers,denschub,ntim,whimboo,sparky
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.

To produce this patch I did all of the following:

1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.

2. Run ./mach lint --linter black --fix

3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.

4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.

5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).

# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D94045
2020-10-23 20:40:42 +00:00
Dorel Luca
95b44c982f Backed out changeset 7558c8821a07 (bug 1654103) for multiple failures. CLOSED TREE 2020-10-22 03:51:06 +03:00
Ricky Stewart
43baed3c18 Bug 1654103: Standardize on Black for Python code in mozilla-central. r=remote-protocol-reviewers,marionette-reviewers,webdriver-reviewers,perftest-reviewers,devtools-backward-compat-reviewers,jgilbert,preferences-reviewers,sylvestre,maja_zf,webcompat-reviewers,denschub,ntim,whimboo,sparky
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.

To produce this patch I did all of the following:

1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.

2. Run ./mach lint --linter black --fix

3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.

4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).

# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D94045
2020-10-21 21:27:27 +00:00
Mike Hommey
feab717fe4 Bug 1661589 - Track decision task duration on perfherder. r=jmaher
Differential Revision: https://phabricator.services.mozilla.com/D88495
2020-08-28 10:45:23 +00:00
Ricky Stewart
34db6aa3b6 Bug 1657650 - Require that Mach command providers subclass MachCommandBase. r=remote-protocol-reviewers,marionette-reviewers,maja_zf,mhentges,froydnj
Today we don't require that `mach` `CommandProvider`s subclass from any particular parent class and we're very lax about the requirements they must meet. While that's convenient in certain circumstances, it has some unfortunate implications for feature development.

Today the only requirements that we have for `CommandProvider`s are that they have an `__init__()` method that takes either 1 or 2 arguments, the second of which must be called `context` and is populated with the `mach` `CommandContext`. Again, while this flexibility is occasionally convenient, it is limiting. As we add features to `mach`, having a better idea what the shape of our `CommandProvider`s are and how we can instantiate them and use them is increasingly important, and this gives us additional control when having `mach` configure `CommandProvider`s based on data that is only available at the `mach` level. In particular, we plan to leverage this in bugs 985141 and 1654074.

Here we add validation to the `CommandProvider` decorator to ensure all classes inherit from `MachCommandBase`, update all `CommandProvider`s in-tree to inherit from `MachCommandBase`, and update source and test code accordingly.

Follow-up work: we now require (de facto) that the `context` be populated with a `topdir` attribute by the `populate_context_handler` function, since instantiating the `MachCommandBase` requires a `topdir` be provided. This is fine for now in the interest of keeping this patch reasonably sized, but some additional refactoring could make this cleaner.

Differential Revision: https://phabricator.services.mozilla.com/D86255
2020-08-07 18:24:59 +00:00
Tom Prince
2b91d423d4 No bug: [taskgrah] Fix mach taskgraph full --tasks <...> on python 3; r=Callek DONTBUILD
Differential Revision: https://phabricator.services.mozilla.com/D85895
2020-08-04 17:16:15 +00:00
Tom Prince
e7d8f225cb No bug: [taskgraph] Make verify_test_packaging handle --target-kind gracefully; r=Callek
If we are generating only a part of the graph, to given kind, don't fail if a
build is packaging tests and there is no corresponding test task, as the tests
may not have been generated.

Differential Revision: https://phabricator.services.mozilla.com/D82097
2020-07-14 07:17:26 +00:00
Tom Prince
1f508bef86 No bug: Update links to hg.mozilla.org to use default instead of tip in taskcluster/; r=Callek
The latter sometimes bites us if we have a legacy support branch that gets
pushed more recently than default.

Differential Revision: https://phabricator.services.mozilla.com/D82628
2020-07-08 18:41:17 +00:00
Tom Prince
9b6e494ab1 Bug 1648591: Remove taskgraph based cron implementation; r=aki
Differential Revision: https://phabricator.services.mozilla.com/D81270
2020-07-07 16:04:00 +00:00
Tom Prince
d69a6151ef Bug 1641971: [taskgraph] Pass explicit decision task id to action.json generation; r=Callek
Differential Revision: https://phabricator.services.mozilla.com/D77550
2020-06-03 02:35:05 +00:00
Tom Prince
d648744c0d Bug 1641971: Allow references to decision task via task-reference/artifact-reference; r=Callek
Differential Revision: https://phabricator.services.mozilla.com/D77547
2020-06-03 02:43:34 +00:00
Narcis Beleuzu
982ec4412a Backed out 7 changesets (bug 1641971) for breaking Gecko Decision Task opt . CLOSED TREE
Backed out changeset a039e60a26d2 (bug 1641971)
Backed out changeset e7d4c490c31a (bug 1641971)
Backed out changeset f2322981b294 (bug 1641971)
Backed out changeset d123b08d2c20 (bug 1641971)
Backed out changeset 57772c0b9b3e (bug 1641971)
Backed out changeset 11a887f73dbb (bug 1641971)
Backed out changeset e9c7a59d4bc5 (bug 1641971)
2020-06-03 05:30:51 +03:00
Tom Prince
4b27ad1785 Bug 1641971: [taskgraph] Pass explicit decision task id to action.json generation; r=Callek
Differential Revision: https://phabricator.services.mozilla.com/D77550
2020-06-02 18:23:20 +00:00
Tom Prince
d9be0f48e6 Bug 1641971: Allow references to decision task via task-reference/artifact-reference; r=Callek
Differential Revision: https://phabricator.services.mozilla.com/D77547
2020-06-02 18:50:10 +00:00
Mitchell Hentges
c8db26f9fb Bug 1641962: Defer imports of distutils.util to ensure bootstrap can run r=rstewart,perftest-reviewers,sparky
Differential Revision: https://phabricator.services.mozilla.com/D77516
2020-05-29 20:17:27 +00:00
Chris AtLee
3000e4f731 Bug 1637381: Add support for extracting zst files to mozbuild tooltool r=glandium
Differential Revision: https://phabricator.services.mozilla.com/D75203
2020-05-21 13:28:33 +00:00
Ricky Stewart
02945d3f8a Bug 1524639: [taskgraph] Enforce unicode strings in schemas r=tomprince
Differential Revision: https://phabricator.services.mozilla.com/D18376
2020-01-21 17:12:08 +00:00
Razvan Maries
1b1233c4a4 Backed out changeset a57aed49dc58 (bug 1524639) for beetmover bustages. a=backout 2020-01-16 14:04:31 +02:00
Ricky Stewart
16061a0a81 Bug 1524639: [taskgraph] Enforce unicode strings in schemas r=tomprince
Differential Revision: https://phabricator.services.mozilla.com/D18376
2020-01-15 20:33:20 +00:00
Andrew Halberstadt
ba964787d7 Bug 1568277 - [taskgraph] Add ability to redirect |mach taskgraph|'s output to a file, r=tomprince
Differential Revision: https://phabricator.services.mozilla.com/D42172
2019-08-16 14:27:14 +00:00