There are a few places where we walk commit ancestry looking for things
attached to a specific revision. Because the repository name is attached
to the index path and because a revision can exist in multiple
repositories, we often have to perform N index lookups to find a result
for a specific revision. This is inefficient.
To facilitate faster index lookups by revision, we introduce a new route
that doesn't contain the repository name. In theory, we should be able
to do this globally - for all repos. However, the configuration of
tasks can vary significantly by repo. So e.g. a linux64 build on
"central" is sufficiently different from a linux64 build on "beta" or
"release." For that reason, this commit takes the conservative
approach and only defines a shared route for repositories with a similar
configuration: the "trunk" repositories.
MozReview-Commit-ID: 8rIgUbzW4eL
There are a few places where we walk commit ancestry looking for things
attached to a specific revision. Because the repository name is attached
to the index path and because a revision can exist in multiple
repositories, we often have to perform N index lookups to find a result
for a specific revision. This is inefficient.
To facilitate faster index lookups by revision, we introduce a new route
that doesn't contain the repository name. In theory, we should be able
to do this globally - for all repos. However, the configuration of
tasks can vary significantly by repo. So e.g. a linux64 build on
"central" is sufficiently different from a linux64 build on "beta" or
"release." For that reason, this commit takes the conservative
approach and only defines a shared route for repositories with a similar
configuration: the "trunk" repositories.
MozReview-Commit-ID: 8rIgUbzW4eL
365731510976 (bug 1380391) added index routes for decision tasks by
pushlog id. This is a good idea. The pushlog id is guaranteed to always
be incrementing (except for repos that are periodically reset, which
we don't care about). It is useful to provide strict ordering for
pushes and is simpler for machines to consume and sort than dates.
So let's index all tasks by pushlog id.
MozReview-Commit-ID: BPqx4ARza1c
This is needed before we can upgrade to flake8 3.3.0, as that version starts flagging these errors.
These files were modified by running:
autopep8 --select E305 --in-place -r <dir>
on the affected directories. I did it one dir at a time and verified the result after each.
MozReview-Commit-ID: FmlsfiKIbtr
To date we have variously specified both worker-type and worker-implementation,
often manually coordinated. We also embedded a few awkward assumptions such as
that the native engine only runs on OS X.
But a worker type has one and only one implementation, and that implementation
is stable over time (as changing it would require simultaneous landings on all
trees).
Instead, this change makes worker-type the primary configuration, and derives
both a worker implementation (defining the payload format) and worker OS
(determining what to include in the payload) from that value. The derivation
occurs when deciding how to implement a particular job, where the run_using
functions are distinguished by worker implementation.
The two-part logic to determine how and where to run a test task based on its
platform is combined into a single transform, `set_worker_type`.
This contains some other related changes:
- MOZ_AUTOMATION is set in specific jobs, rather than everywhere docker-worker
is used
- the URL to test packages is factored out into a shared function
- docker-worker test defaults are applied in `mozharness_test.py`
- the WORKER_TYPE array in `task.py`, formerly mixing two types of keys, is
split
- the 'invalid' workerType is assigned an 'invalid' implementation
- all tasks that do not use job descriptions but use docker-worker, etc. have
`worker.os` added
Tested to not produce a substantially different taskgraph for a regular push, a
try push, or a nightly cron.
MozReview-Commit-ID: LDHrmrpBo7I
Tasks should be assigned a priority based on the branch they originated from. It
is important that certain branches receive preferential treatment, such as a release
branch task being executed before a task from Try. Branch priority mirrors
the priorities defined within buildbot.
MozReview-Commit-ID: 8qR9F34lzzc
Tasks should be assigned a priority based on the branch they originated from. It
is important that certain branches receive preferential treatment, such as a release
branch task being executed before a task from Try. Branch priority mirrors
the priorities defined within buildbot.
MozReview-Commit-ID: 8qR9F34lzzc
Docker-worker's `command` field is actually not required, as it will run a
docker image's default command when command is not specified.
MozReview-Commit-ID: I3vBHeixlxW
* add run.using = 'run-task' for native-engine
* modify run-task to run on OS X
- not as root
- without assuming /home/worker (using ~ and os.expanduser instead)
- hg is in /usr/local/bin on OS X; trust the PATH
* add_build_dependency isn't docker-worker specific, so just rename
* support_vcs_checkout modified to omit caches on native-engine
* don't download fingerprints on OS X; these hosts are configured with
the proper fingerprint via puppet
MozReview-Commit-ID: C83XClXtcn4
This also adds a new 'py' group which should help make it obvious which tasks are python
unittests. The tg and MH tasks won't show up in this new py group right away due to a
caching bug in the treeherder database, but eventually they should move over.
MozReview-Commit-ID: JwL1NAR4bnY
