This approach allows to specify the `docker-image` and set of
`toolchains` to the l10n leaf jobs using the `by-platform:` override
mechanism. We don't support anything but in-tree docker images at
this time, and the schema will warn if a different type of docker
configuration block is used. It wouldn't be hard to grow the
additional blocks, but let's reduce duplication for now.
It might be considered better to inherit the `docker-image` and set of
`toolchains` from the underlying `dependent-task`, but we don't do
that for two reasons. The main reason is that it's an explicit goal
to be able to "cross repack": to repack, say, a Windows binary on a
Linux worker. In that situation, the docker-image and toolchains
differ between the builder and the repack worker.
A smaller technical obstruction is that by the time the l10n transform
sees the dependent task, the docker image and set of toolchains have
been processed. The l10n transform would have to "reconstitute" the
docker image changes and the set of toolchains; it would be very
fragile.
Taken together, it's better to be explicit, reduce unexpected
interactions, and repeat the information in the l10n leaf tasks.
MozReview-Commit-ID: TmgJyYU5dx
Diffing `target-graph`s was difficult because the locales kept shuffling.
This patch will keep the locales in alphabetical order.
MozReview-Commit-ID: GvGYF7j9ftq
Stop hardcoding `android`, since we want to use this for desktop too.
We could potentially remove the `android` platform from the bumper configs
at this point.
We strip `-nightly` from the `build_platform` before comparing against the
`l10n-changesets.json` platform list. If we want to support different sets
of ci and nightly locales, we could either:
- point at a second changesets file for ci. This could either be a flatfile
or json; either works. or,
- explicitly name `win32-nightly` etc. in the platform list, and stop removing
the `-nightly` from the `build_platform` before comparing. This means some
locales may have up to 10 different platforms listed. This may get unwieldy,
but would be explicit.
MozReview-Commit-ID: Fvpby92cXdg
The signing pool is a limited resource. We already have logic to avoid re-signing
widevine if the .sig files are there, and we're taking advantage of that on linux
and windows. OSX was the lone desktop os still pointing at the unsigned bits for
l10n.
We're pointing at repackage instead of signing because the l10n logic expects a dmg.
MozReview-Commit-ID: FtWa4o5s7aJ
This is needed before we can upgrade to flake8 3.3.0, as that version starts flagging these errors.
These files were modified by running:
autopep8 --select E305 --in-place -r <dir>
on the affected directories. I did it one dir at a time and verified the result after each.
MozReview-Commit-ID: FmlsfiKIbtr
To date we have variously specified both worker-type and worker-implementation,
often manually coordinated. We also embedded a few awkward assumptions such as
that the native engine only runs on OS X.
But a worker type has one and only one implementation, and that implementation
is stable over time (as changing it would require simultaneous landings on all
trees).
Instead, this change makes worker-type the primary configuration, and derives
both a worker implementation (defining the payload format) and worker OS
(determining what to include in the payload) from that value. The derivation
occurs when deciding how to implement a particular job, where the run_using
functions are distinguished by worker implementation.
The two-part logic to determine how and where to run a test task based on its
platform is combined into a single transform, `set_worker_type`.
This contains some other related changes:
- MOZ_AUTOMATION is set in specific jobs, rather than everywhere docker-worker
is used
- the URL to test packages is factored out into a shared function
- docker-worker test defaults are applied in `mozharness_test.py`
- the WORKER_TYPE array in `task.py`, formerly mixing two types of keys, is
split
- the 'invalid' workerType is assigned an 'invalid' implementation
- all tasks that do not use job descriptions but use docker-worker, etc. have
`worker.os` added
Tested to not produce a substantially different taskgraph for a regular push, a
try push, or a nightly cron.
MozReview-Commit-ID: LDHrmrpBo7I
Adds l10n and nightly indexing, matching (better) what Buildbot is currently doing with these types of tasks
(This patch is against `date`, will be grafted on review for real landing, using autoland)
MozReview-Commit-ID: K0BYwaCm6xL
