Bug 1359976: base worker payload generation on worker-type; r=wcosta r=aki

To date we have variously specified both worker-type and worker-implementation,
often manually coordinated. We also embedded a few awkward assumptions such as
that the native engine only runs on OS X.

But a worker type has one and only one implementation, and that implementation
is stable over time (as changing it would require simultaneous landings on all
trees).

Instead, this change makes worker-type the primary configuration, and derives
both a worker implementation (defining the payload format) and worker OS
(determining what to include in the payload) from that value. The derivation
occurs when deciding how to implement a particular job, where the run_using
functions are distinguished by worker implementation.

The two-part logic to determine how and where to run a test task based on its
platform is combined into a single transform, `set_worker_type`.

This contains some other related changes:
 - MOZ_AUTOMATION is set in specific jobs, rather than everywhere docker-worker
   is used
 - the URL to test packages is factored out into a shared function
 - docker-worker test defaults are applied in `mozharness_test.py`
 - the WORKER_TYPE array in `task.py`, formerly mixing two types of keys, is
   split
 - the 'invalid' workerType is assigned an 'invalid' implementation
 - all tasks that do not use job descriptions but use docker-worker, etc. have
   `worker.os` added

Tested to not produce a substantially different taskgraph for a regular push, a
try push, or a nightly cron.

MozReview-Commit-ID: LDHrmrpBo7I

taskcluster/taskgraph/transforms/task.py

@@ -154,6 +154,7 @@ task_description_schema = Schema({
     # information specific to the worker implementation that will run this task
     'worker': Any({
         Required('implementation'): Any('docker-worker', 'docker-engine'),
+        Required('os'): 'linux',
 
         # For tasks that will run in docker-worker or docker-engine, this is the
         # name of the docker image or in-tree docker image to run the task in.  If
@@ -215,9 +216,10 @@ task_description_schema = Schema({
         Optional('retry-exit-status'): int,
 
     }, {
+        Required('implementation'): 'generic-worker',
+        Required('os'): Any('windows', 'macosx'),
         # see http://schemas.taskcluster.net/generic-worker/v1/payload.json
         # and https://docs.taskcluster.net/reference/workers/generic-worker/payload
-        Required('implementation'): 'generic-worker',
 
         # command is a list of commands to run, sequentially
         # on Windows, each command is a string, on OS X and Linux, each command is
@@ -308,6 +310,7 @@ task_description_schema = Schema({
         },
     }, {
         Required('implementation'): 'native-engine',
+        Required('os'): Any('macosx', 'linux'),
 
         # A link for an executable to download
         Optional('context'): basestring,
@@ -396,6 +399,12 @@ task_description_schema = Schema({
         Required('implementation'): 'push-apk-breakpoint',
         Required('payload'): object,
 
+    }, {
+        Required('implementation'): 'invalid',
+        # an invalid task is one which should never actually be created; this is used in
+        # release automation on branches where the task just doesn't make sense
+        Extra: object,
+
     }, {
         Required('implementation'): 'push-apk',
 
@@ -741,6 +750,11 @@ def build_push_apk_breakpoint_payload(config, task, task_def):
     task_def['payload'] = task['worker']['payload']
 
 
+@payload_builder('invalid')
+def build_invalid_payload(config, task, task_def):
+    task_def['payload'] = 'invalid task - should never be created'
+
+
 @payload_builder('native-engine')
 def build_macosx_engine_payload(config, task, task_def):
     worker = task['worker']