Bug 1245953 - Support defining non-build/test Task Cluster tasks; r=garndt

Currently, tasks are either "build" or "test" tasks. And "test" tasks
are dependent on "build" tasks, so they are effectively an extension of
"build" tasks.

Not everything is a "build" task. Not everything is associated with a
specific platform.

This commit introduces support for defining non-build "tasks" under the
"tasks" top-level element of a jobs YAML file. Interally, they are
treated as "build" tasks but are declared differently.

By default, all these tasks run.

The -j/--job argument has been added to the try syntax parser. It
specifies an opt-in list of these non-build tasks to run. By default, it
runs all of them.

The eslint-gecko "build" task has been moved to this new mechanism.

Documentation for the new task type have been added.

There is definitely some wonkiness in this implementation. For example,
there are references to "build_name," "build_type," and "build_product,"
which arguably are no longer relevant to generic tasks. However, they
appear to be so integrated into task processing (including route names)
that I'm a bit scared to change them.

MozReview-Commit-ID: BY219tLFb6Z

testing/taskcluster/docs/index.rst

@@ -145,6 +145,32 @@ chunk
 total_chunks
    Total number of chunks
 
+Generic Tasks
+=============
+
+Generic tasks are neither build tasks nor test tasks. They are intended for
+tasks that don't fit into either category.
+
+.. important::
+
+   Generic tasks are a new feature and still under development. The
+   conventions will likely change significantly.
+
+Generic tasks are defined under a top-level ``tasks`` dictionary in the
+YAML. Keys in the dictionary are the unique task name. Values are
+dictionaries of task attributes. The following attributes can be defined:
+
+task
+   *required* Path to the YAML file declaring the task.
+
+root
+   *optional* Boolean indicating whether this is a *root* task. Root
+   tasks are scheduled immediately, if scheduled to run.
+
+additional-parameters
+   *optional* Dictionary of additional parameters to pass to template
+   expansion.
+
 Developing
 ==========
 

testing/taskcluster/taskcluster_graph/commit_parser.py

@@ -258,13 +258,25 @@ def parse_commit(message, jobs):
     parser.add_argument('-p', '--platform', nargs='?', dest='platforms', const='all', default='all')
     parser.add_argument('-u', '--unittests', nargs='?', dest='tests', const='all', default='all')
     parser.add_argument('-i', '--interactive', dest='interactive', action='store_true', default=False)
+    parser.add_argument('-j', '--job', dest='jobs', action='append')
     # In order to run test jobs multiple times
     parser.add_argument('--trigger-tests', dest='trigger_tests', type=int, default=1)
     args, unknown = parser.parse_known_args(parts[try_idx:])
 
+    # Normalize default value to something easier to detect.
+    if args.jobs == ['all']:
+        args.jobs = None
+
+    # Expand commas.
+    if args.jobs:
+        expanded = []
+        for job in args.jobs:
+            expanded.extend(j.strip() for j in job.split(','))
+        args.jobs = expanded
+
     # Then builds...
     if args.build_types is None:
-        return []
+        args.build_types = []
 
     build_types = [ BUILD_TYPE_ALIASES.get(build_type, build_type) for
             build_type in args.build_types ]
@@ -319,6 +331,27 @@ def parse_commit(message, jobs):
                 'interactive': args.interactive,
             })
 
+    # Process miscellaneous tasks.
+    for name, task in sorted(jobs.get('tasks', {}).items()):
+        # args.jobs == None implies all tasks.
+        if args.jobs is not None and name not in args.jobs:
+            continue
+
+        # TODO support tasks that are defined as dependent on another one.
+        if not task.get('root', False):
+            continue
+
+        result.append({
+            'task': task['task'],
+            'post-build': [],
+            'dependents': [],
+            'additional-parameters': task.get('additional-parameters', {}),
+            'build_name': name,
+            # TODO support declaring a different build type
+            'build_type': name,
+            'interactive': args.interactive,
+        })
+
     # Times that test jobs will be scheduled
     trigger_tests = args.trigger_tests
 

testing/taskcluster/tasks/branches/base_job_flags.yml

@@ -106,7 +106,6 @@ flags:
     - linux64-st-an
     - macosx64
     - macosx64-st-an
-    - eslint-gecko
 
   tests:
     - cppunit

testing/taskcluster/tasks/branches/base_jobs.yml

@@ -168,12 +168,6 @@ builds:
     types:
       opt:
         task: tasks/builds/android_api_15_b2gdroid.yml
-  eslint-gecko:
-    platforms:
-      - lint
-    types:
-      opt:
-        task: tasks/tests/eslint-gecko.yml
 
 tests:
   cppunit:
@@ -310,3 +304,9 @@ tests:
         task: tasks/tests/b2g_emulator_xpcshell_chunked.yml
       tasks/builds/dbg_linux64.yml:
         task: tasks/tests/fx_linux64_xpcshell.yml
+
+# Miscellaneous tasks.
+tasks:
+  eslint-gecko:
+    task: tasks/tests/eslint-gecko.yml
+    root: true

testing/taskcluster/tasks/lint.yml

@@ -18,8 +18,8 @@ task:
   schedulerId: task-graph-scheduler
 
   routes:
-    - 'index.gecko.v1.{{project}}.revision.linux.{{head_rev}}.{{build_name}}.{{build_type}}'
-    - 'index.gecko.v1.{{project}}.latest.linux.{{build_name}}.{{build_type}}'
+    - 'index.gecko.v1.{{project}}.revision.linux.{{head_rev}}.{{build_name}}'
+    - 'index.gecko.v1.{{project}}.latest.linux.{{build_name}}'
   scopes:
     # Nearly all of our build tasks use tc-vcs so just include the scope across
     # the board.
@@ -34,7 +34,7 @@ task:
 
   extra:
     build_product: '{{build_product}}'
-    build_name: '{{build_name}}'
-    build_type: '{{build_type}}'
+    build_name: eslint-gecko
+    build_type: opt
     index:
       rank: {{pushlog_id}}