This adds some new optimization strategies. For tests, we use Either(SETA,
SkipUnlessSchedules), thereby giving both mechanisms a chance to skip tasks. On
try, SETA is omitted.
MozReview-Commit-ID: GL4tlwyeBa6
In preparation for much more thorough optimization of task-graphs, this
makes a few changes:
* optimization is split into thre phases, with task removal in one phase
(following dependency links) and task replacement in the next (in the
reverse order).
* optimization uses class instances instead of functions for optimizations;
this allows different functions for different phases, and also leaves open
the possibility of composing optimizations.
* the replacement phase can also support removal; this is when utility tasks
like symbol uploads can be optimized away iff their parent task is
optimized.
MozReview-Commit-ID: C5QznNpwqXn
It is not at *all* clear how multiple optimizations for a single task should
interact. No simple logical operation is right in all cases, and in fact in
most imaginable cases the desired behavior turns out to be independent of all
but one of the optimizations. For example, given both `seta` and
`skip-unless-files-changed` optimizations, if SETA says to skip a test, it is
low value and should be skipped regardless of what files have changed. But if
SETA says to run a test, then it has likely been skipped in previous pushes, so
it should be run regardless of what has changed in this push.
This also adds a bit more output about optimization, that may be useful for
anyone wondering why a particular job didn't run.
MozReview-Commit-ID: 3OsvRnWjai4
This adds some new optimization strategies. For tests, we use Either(SETA,
SkipUnlessSchedules), thereby giving both mechanisms a chance to skip tasks. On
try, SETA is omitted.
MozReview-Commit-ID: GL4tlwyeBa6
In preparation for much more thorough optimization of task-graphs, this
makes a few changes:
* optimization is split into thre phases, with task removal in one phase
(following dependency links) and task replacement in the next (in the
reverse order).
* optimization uses class instances instead of functions for optimizations;
this allows different functions for different phases, and also leaves open
the possibility of composing optimizations.
* the replacement phase can also support removal; this is when utility tasks
like symbol uploads can be optimized away iff their parent task is
optimized.
MozReview-Commit-ID: C5QznNpwqXn
It is not at *all* clear how multiple optimizations for a single task should
interact. No simple logical operation is right in all cases, and in fact in
most imaginable cases the desired behavior turns out to be independent of all
but one of the optimizations. For example, given both `seta` and
`skip-unless-files-changed` optimizations, if SETA says to skip a test, it is
low value and should be skipped regardless of what files have changed. But if
SETA says to run a test, then it has likely been skipped in previous pushes, so
it should be run regardless of what has changed in this push.
This also adds a bit more output about optimization, that may be useful for
anyone wondering why a particular job didn't run.
MozReview-Commit-ID: 3OsvRnWjai4
This adds some new optimization strategies. For tests, we use Either(SETA,
SkipUnlessSchedules), thereby giving both mechanisms a chance to skip tasks. On
try, SETA is omitted.
MozReview-Commit-ID: GL4tlwyeBa6
In preparation for much more thorough optimization of task-graphs, this
makes a few changes:
* optimization is split into thre phases, with task removal in one phase
(following dependency links) and task replacement in the next (in the
reverse order).
* optimization uses class instances instead of functions for optimizations;
this allows different functions for different phases, and also leaves open
the possibility of composing optimizations.
* the replacement phase can also support removal; this is when utility tasks
like symbol uploads can be optimized away iff their parent task is
optimized.
MozReview-Commit-ID: C5QznNpwqXn
It is not at *all* clear how multiple optimizations for a single task should
interact. No simple logical operation is right in all cases, and in fact in
most imaginable cases the desired behavior turns out to be independent of all
but one of the optimizations. For example, given both `seta` and
`skip-unless-files-changed` optimizations, if SETA says to skip a test, it is
low value and should be skipped regardless of what files have changed. But if
SETA says to run a test, then it has likely been skipped in previous pushes, so
it should be run regardless of what has changed in this push.
This also adds a bit more output about optimization, that may be useful for
anyone wondering why a particular job didn't run.
MozReview-Commit-ID: 3OsvRnWjai4
These both have a similar form, recursing over a nested JSON structure, and are
useful outside of the modules in which they are defined.
MozReview-Commit-ID: 1bsRtlaQol7
This simplifies the return value of optimize functions:
* False -- don't optimize
* True -- optimize away
* <taskId> -- replace with this task
Original patch by ayodeji.oyewole, with test updates by dustin.
MozReview-Commit-ID: HKoWcINVSnV
MikeLing initially did this in bug 1287018. The intent of this conditional was
to make optimization faster by not even checking most tasks, based on the
assumption that if the prerequisite to a task has changed (for example, a
docker image or a build), then naturally we will want to execute that task.
However, as we have developed actual optimization methods, this has proven not
to be the case: we might want to optimize a test out if its inputs have not
changed, even if a new installer has been built. Similarly, SETA may optimize
tasks out even if their inputs have changed.
MozReview-Commit-ID: LgHET3Z84GB
* Implement & document optimization (although legacy kind doesn't do much of it)
* Introduce `optimize_target_tasks` parameter to control whether tasks in the
target set can be optimized (no for try, yes for most other branches)
* Refactor to include resolved taskIds in the optimized task graph
* Include a `label-to-taskid.json` artifact.
* Introduce {'task-reference': '... <dependency-name> ...'} for referring to
parent tasks' taskId.
MozReview-Commit-ID: LWvlWNz49U5
