Commit Graph

79 Commits

Author SHA1 Message Date
Dustin J. Mitchell
935b8b7768 Bug 1403342 - default to -j none and do not optimize_target_tasks for try; r=ahal
With this in place, all `-j`obs will not run by default on try.  This will omit
such jobs in most try pushes even if files-changed matches. This is
unfortunate, but better than running them unconditionally.  Fuzzy selections,
and later `just try it` pushes, are the ultimate solution here.

With this change, a push with no try syntax or try_task_config.json will schedule
no tasks at all.

MozReview-Commit-ID: FGjqlDW1FT6
2017-10-03 21:15:15 +00:00
Dustin J. Mitchell
98b1397547 Bug 1403342 - only set try-related parameters on the try branch; r=ahal
Specifically, this avoids setting optimize_target_tasks to True unconditionally
for non-try branches.

MozReview-Commit-ID: HSJFLmqbMmZ
2017-10-02 19:13:22 +00:00
Tom Prince
25e12b05a3 Bug 1385055 - Add some taskgraph parameters for dealing with comm-central and related repositories. r=dustin
MozReview-Commit-ID: 4tdxfJcx8Fz
2017-07-27 12:26:48 -06:00
Brian Stack
666b3098bc Bug 1406177 - Remove action.yml now that it is unused r=garndt
MozReview-Commit-ID: IIX125xFoMx
2017-10-05 14:47:15 -07:00
Andrew Halberstadt
0bb75a6263 Bug 1401199 - [taskgraph] Use default parameter values when strict=False, r=dustin
MozReview-Commit-ID: 9XWlLeGcPeQ
2017-09-29 11:35:30 -04:00
Dustin J. Mitchell
d08f492fb6 Bug 1383880: parse try config during the decision task; r=ahal
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.

The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.

MozReview-Commit-ID: AXJEGLh6pEV
2017-08-21 23:14:14 +00:00
Phil Ringnalda
b76b48b887 Backed out 12 changesets (bug 1383880) for decision task bustage
CLOSED TREE

Backed out changeset fd3615e7e0a3 (bug 1383880)
Backed out changeset 8cceb6a82bfb (bug 1383880)
Backed out changeset 571a6c9054a5 (bug 1383880)
Backed out changeset ec2b8ba5a949 (bug 1383880)
Backed out changeset 8e5847d9acda (bug 1383880)
Backed out changeset b354fdf6e233 (bug 1383880)
Backed out changeset ebdd6ccbcfca (bug 1383880)
Backed out changeset ebcc9d20981a (bug 1383880)
Backed out changeset 97eedc84d6e8 (bug 1383880)
Backed out changeset a3116da52b4e (bug 1383880)
Backed out changeset b3eb0c939720 (bug 1383880)
Backed out changeset 7c07cb798530 (bug 1383880)

MozReview-Commit-ID: EPDuQHr7w2y
2017-09-20 19:57:39 -07:00
Dustin J. Mitchell
a7457a95f7 Bug 1383880: parse try config during the decision task; r=ahal
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.

The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.

MozReview-Commit-ID: AXJEGLh6pEV
2017-08-21 23:14:14 +00:00
Simon Fraser
0ab9dada04 Bug 1342392 Migrate partial update generation in-tree r=rail
MozReview-Commit-ID: G6EFXY0UzSa
2017-09-18 13:36:36 +01:00
Sebastian Hengst
f1bbda3eeb Backed out changeset 0d71c0be62de (bug 1342392) for bustage in partial update generator for docker images: recompress.sh not found. r=backout 2017-09-18 14:15:42 +02:00
Simon Fraser
d2b8ac4d0d Bug 1342392 Migrate partial update generation in-tree r=rail
MozReview-Commit-ID: 84fP48sMYhS
2017-09-15 11:04:35 +01:00
Sebastian Hengst
ca434b9d52 Backed out changeset a43387ee65d7 (bug 1342392) for failing flake8 and py-compat jobs. r=backout 2017-09-18 13:50:16 +02:00
Simon Fraser
498d6b87a7 Bug 1342392 Migrate partial update generation in-tree r=rail
MozReview-Commit-ID: 84fP48sMYhS
2017-09-15 11:04:35 +01:00
Wes Kocher
d62fefa0af Backed out 2 changesets (bug 1342392) for bustage a=backout
Backed out changeset b0e86286b173 (bug 1342392)
Backed out changeset 01bf0c29331e (bug 1342392)

MozReview-Commit-ID: 71fcOZ2RgEp
2017-09-11 17:35:09 -07:00
Simon Fraser
278648e798 Bug 1342392 Move partial update generation in-tree r=dustin,jlorenzo 2017-09-11 09:55:52 +01:00
Sebastian Hengst
27461f48cf Backed out 2 changesets (bug 1342392) for adding .orig file and flake lint failures. r=backout on a CLOSED TREE
Backed out changeset bf0bcdc7482e (bug 1342392)
Backed out changeset 7f5ae5ff1181 (bug 1342392)
2017-09-11 10:38:08 +02:00
Simon Fraser
e9d9241564 Bug 1342392 Move partial update generation in-tree r=dustin,jlorenzo 2017-09-11 09:24:10 +01:00
Sebastian Hengst
cd4005d4a7 Backed out changeset b53ff084c2d7 (bug 1383880) 2017-09-06 17:47:45 +02:00
Dustin J. Mitchell
40fa2c8f48 Bug 1383880: parse try config during the decision task; r=ahal
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.

The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.

MozReview-Commit-ID: AXJEGLh6pEV
2017-08-21 23:14:14 +00:00
Ryan VanderMeulen
c41dc43ff0 Backed out 9 changesets (bug 1383880) for decision task bustage.
Backed out changeset 53f5d47a7cb0 (bug 1383880)
Backed out changeset a0abda41172a (bug 1383880)
Backed out changeset 729a7e2091e8 (bug 1383880)
Backed out changeset a33f5a14a471 (bug 1383880)
Backed out changeset 5b10d321cfee (bug 1383880)
Backed out changeset 8056488d8aed (bug 1383880)
Backed out changeset e62c90e3c1e8 (bug 1383880)
Backed out changeset 91f116ce6c2a (bug 1383880)
Backed out changeset 045498bc36c4 (bug 1383880)
2017-09-01 15:37:08 -04:00
Dustin J. Mitchell
464d6eeb31 Bug 1383880: parse try config during the decision task; r=ahal
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.

The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.

MozReview-Commit-ID: AXJEGLh6pEV
2017-08-21 23:14:14 +00:00
Dustin J. Mitchell
c73a5bb8f8 Bug 1391776: cleanup of taskgraph docs; r=ahal
* eliminate heading for test kinds, of which there is now only one
* make the caches document have a single heading in the TOC
* break out mach commands into a separate document, add ./mach taskgraph morphed
* remove docs for YAML templates support (the .yml file wasn't actually
  used -- I expect it was a merge leftover); these are still used for actions.yml,
  but once that is gone the code should be removed, too.
* break try out into its own document, edit to distinguish "how to run try"
  from "how to generate config"

MozReview-Commit-ID: 76ZopWA9TPL
2017-08-23 15:22:10 -04:00
Andrew Halberstadt
91fd656da8 Bug 1387135 - Add ability to apply templates to task definitions via try_task_config.json, r=dustin
This provides a mechanism to modify the behaviour of tasks from a try push. The try_task_config.json
looks something like:

{
  "tasks": ["build-linux64/opt", "test-linux64/opt-mochitest-e10s-1"],
  "templates": {
    "artifact": {"enabled": 1}
  }
}

This tells taskgraph to apply the 'artifact' template to all tasks. Templates are JSONe based
.yml files that live under taskcluster/taskgraph/templates. Taskgraph will render every template
against every task definition. The templates themselves can then use JSONe condition statements to
filter out which tasks they should or shouldn't apply to.

MozReview-Commit-ID: J8HVZzOt4mX
2017-08-15 11:36:29 -04:00
Sebastian Hengst
0f53503b6b merge mozilla-inbound to mozilla-central. r=merge a=merge
MozReview-Commit-ID: 44WBcWjnVo
2017-07-30 11:19:17 +02:00
Andrew Halberstadt
9cbe59ace4 Bug 1380306 - Create a new 'try_task_config' method for scheduling tasks on try, r=dustin
This introduces a 'try_task_config' method of scheduling. En lieu of (or in addition to) try
syntax, you can now check in a file called 'try_task_config.json' to the root of the source
tree. The format is either a list of task labels, or dict where task labels are the keys.
Taskcluster will simply schedule any tasks that are listed there.

This file is primarily meant to be generated by tools (which don't exist yet), as the json
format is much easier for tools to generate or consume. These tools should use an in-memory
commit to add the file so it is automatically removed again after the push.

A server-side hook will be added in bug 1380357 to prevent this file from accidentally
landing on non-try trees.

MozReview-Commit-ID: 2zKfZXuuDhH
2017-06-27 13:33:20 -07:00
Wes Kocher
0677a423fa Backed out changeset 9368bd804513 (bug 1384729) for build failures in test_mozbuild_reading.py a=backout CLOSED TREE
MozReview-Commit-ID: 2etJcpEm0ye
2017-07-28 11:45:38 -07:00
Wes Kocher
3a26256f42 Backed out 3 changesets (bug 1385151, bug 1385097, bug 1384729) because more needed to be backed out
Backed out changeset 078c33d62327 (bug 1384729)
Backed out changeset 499ff142be61 (bug 1385097)
Backed out changeset c0e98ba118f1 (bug 1385151)

MozReview-Commit-ID: 5kOPUORK6hO
2017-07-28 11:41:36 -07:00
Sebastian Hengst
829ab4e251 Backed out changeset 9368bd804513 (bug 1384729) for bustage: pattern 'actions/**' in 'taskcluster/moz.build' corresponds to no files in the tree. r=backout on a CLOSED TREE 2017-07-28 20:36:25 +02:00
Brian Stack
020215e823 Bug 1384729 - Move actions into taskgraph dir r=dustin
MozReview-Commit-ID: HK5MR8Xw3On
2017-07-27 10:47:37 -07:00
Brian Stack
5d5a9cecb1 Bug 1384729 - Move actions into taskgraph dir r=dustin
MozReview-Commit-ID: HK5MR8Xw3On
2017-07-27 10:47:37 -07:00
Paul Bignier
bd0dace365 Bug 1354927 - Typo fix in taskcluster. r=MattN 2017-04-09 11:48:00 -04:00
Dustin J. Mitchell
9035033b90 Bug 1351011: remove unused triggered_by property; r=Callek
The purpose of this parameter has been superseded by the `include_nightly`
property.

MozReview-Commit-ID: 4iXQsv9Drqg
2017-03-27 20:10:08 +00:00
Kim Moir
f67722b50b Bug 1339542 - When Servo vcs sync service lands Servo commits in autoland repo, run linux64-stylo tests but skip other platforms (to reduce test load) r=dustin DONTBUILD 2017-04-03 15:48:09 -04:00
Wes Kocher
650b2cfe08 Merge m-c to inbound, a=merge
MozReview-Commit-ID: AzYu7EXTIlg
2017-03-14 17:23:26 -07:00
Aki Sasaki
2a0d264176 bug 1344321 - fix flake8. r=bustage a=release DONTBUILD 2017-03-14 16:39:03 -07:00
Aki Sasaki
82ede80faa bug 1344321 - enable run-on-projects for nightlies. r=dustin a=release
This patch enables `run-on-projects` to work appropriately for
nightly builds and tests.  Initially, we were setting an empty
`run-on-projects` for nightly `build_platform`s, then explicitly
targeting the platforms in nightly-specific `target_task_method`s.

Instead, this patch enables nightlies to `run-on-projects` everywhere,
but governs the use of nightlies by either the `include_nightly`
parameter, or the `--include-nightly` try option.  This lets us filter
nightly-related `target_task_method`s against `run-on-projects` without
losing all nightly tasks.

Then, enable spidermonkey tests by removing optimization from beta and
release.  This patch also enables everything then disables specific
tasks, rather than disabling everything and enabling specific tasks.
Since we're beginning with a `filter_for_project` call, we should be
able to reduce these if blocks to zero over time, if desired.

MozReview-Commit-ID: A9tolynaChF
2017-03-14 15:31:58 -07:00
Dustin J. Mitchell
cef5efbeff Bug 1333255: introduce graph morphs, use them to make index tasks; r=jonasfj
Graph morphs modify the graph after optimization, without changing its meaning.
In this case, that means adding index tasks that will insert paths into the
index beyond the relatively limited number afforded in task.routes.

MozReview-Commit-ID: AJy4exX7q2v
2017-03-07 20:39:27 +00:00
Aki Sasaki
3f38919567 bug 1344321 - add nightly test support. r=dustin a=release
this patch:

- adds linux{32,64}-nightly/opt test platforms that mirror the non-nightly test platforms.

- adds an `include_nightly` per-project parameter; this is refered to in the default `target_task_method`.  It's still possible to launch custom `target_task_method`s to trigger nightlies against, say, try.

- adds a `filter_for_project` method in `target_tasks.py` that allows for `include_nightly` and `run_on_projects` filtering in the various `target_task_method`s.

- adds nightly filtering into the `TryOptionSyntax` object.  By default, this will be off.  To trigger nightly tests on try, either submit a new decision task with a different `target_task_method` (e.g. `nightly_fennec`) or flip the `include_nightly` flag to True.

- adds the `nightly` attribute to tests if their builds have that attribute.

MozReview-Commit-ID: DttIZH0BHS2
2017-03-09 16:32:41 -08:00
Mike Hommey
32f26de395 Bug 1341214 - Define GECKO in a single location. r=dustin
Instead of every file trying to get the top source directory having an
ad-hoc definition that gets wrong if the files gets moved around for
some reason, define it in a more central location.
2017-02-17 11:56:12 +09:00
Alin Selagea
072571c24f Bug 1339179 - Set up pine for activity-stream builds & tests, r=kmoir 2017-02-24 17:19:04 +02:00
Wes Kocher
d1b8cb0d97 Merge m-c to inbound, a=merge
MozReview-Commit-ID: HDHJC8X9xWG
2017-02-22 15:06:46 -08:00
Aki Sasaki
61096c536f bug 1340609 - promotable beta linux builds on push. r=dustin a=release
MozReview-Commit-ID: JAqifpgD3vW
2017-02-22 10:24:58 -08:00
Carsten "Tomcat" Book
20a26945dc Backed out changeset d8122277e9fd (bug 1341214) for causing on m-c problems in vp tc tasks 2017-02-22 15:33:42 +01:00
Mike Hommey
eb5916da95 Bug 1341214 - Define GECKO in a single location. r=dustin
Instead of every file trying to get the top source directory having an
ad-hoc definition that gets wrong if the files gets moved around for
some reason, define it in a more central location.
2017-02-17 11:56:12 +09:00
Gregory Szorc
71072181cc Bug 1336607 - Remove dummy geckolib and taskgraph filter; r=froydnj
Now that we use the real geckolib and have all dependencies vendored,
the dummy geckolib is no longer required, so we remove it.

Also, the taskgraph code for testing for Servo's presence always
passes and is no longer needed, so we remove it.

Pushed on a CLOSED TREE because ¯\_(ツ)_/¯

MozReview-Commit-ID: ITAqArK4Bks
2017-02-03 16:23:14 -08:00
Jonas Finnemann Jensen
c8c1bd5578 Bug 1332506 - Treeherder actions defined in-tree. r=dustin
Framework for defining actions in-tree that can be displayed
and triggered from Treeherder.

MozReview-Commit-ID: 3rvwgy2i4xu
2017-01-31 15:34:05 -08:00
Brian Stack
d28e91043e Bug 1325479 - Fix repo scopes in action task r=dustin
MozReview-Commit-ID: AGfoFtjR5LI
2016-12-22 17:08:33 -05:00
Brian Stack
18ccebf6e8 Bug 1324094 - Add correct cache scopes to action tasks r=armenzg,dustin
MozReview-Commit-ID: KxtG4cHU0Mm
2016-12-16 13:06:57 -08:00
Brian Stack
5e07795db3 Bug 1289823 - Add backfilling as an action-task r=armenzg,dustin
MozReview-Commit-ID: HALwE6Q0Lch
2016-12-07 13:33:20 -10:00
Brian Stack
37e9b8f820 Bug 1289823 - Make taskcluster action-task more flexible r=armenzg,dustin
MozReview-Commit-ID: 8qoJsFeFm6I
2016-11-28 14:15:31 -08:00