corysanin/tubestation
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Backed out 5 changesets (bug 1373326) for breaking the decision task a=backout

Browse Source 
Backed out changeset 7f0177a71077 (bug 1373326)
Backed out changeset a678d4b0ee78 (bug 1373326)
Backed out changeset 2fdc6c2ad384 (bug 1373326)
Backed out changeset 97b785a31a3c (bug 1373326)
Backed out changeset a5fcfe57e6d4 (bug 1373326)

MozReview-Commit-ID: 9kcJzI1hFja
This commit is contained in:
Wes Kocher
2017-06-20 17:43:37 -07:00
parent 7e3f16609d
commit 882dffad8c
19 changed files with 140 additions and 245 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
taskcluster/ci/balrog/kind.yml
View File

@@ -12,7 +12,6 @@ kind-dependencies:
- beetmover
- beetmover-l10n
- beetmover-repackage
- beetmover-repackage-l10n
only-for-attributes:
- nightly

3
taskcluster/ci/beetmover-l10n/kind.yml
View File

@@ -14,6 +14,3 @@ kind-dependencies:
only-for-attributes:
- nightly
not-for-build-platforms:
- macosx64-nightly/opt

16
taskcluster/ci/beetmover-repackage-l10n/kind.yml
View File

@@ -1,16 +0,0 @@
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
loader: taskgraph.loader.single_dep:loader
transforms:
- taskgraph.transforms.beetmover_repackage_l10n:transforms
- taskgraph.transforms.beetmover_repackage:transforms
- taskgraph.transforms.task:transforms
kind-dependencies:
- repackage-l10n
only-for-build-platforms:
- macosx64-nightly/opt

1
taskcluster/ci/checksums-signing/kind.yml
View File

@@ -12,7 +12,6 @@ kind-dependencies:
- beetmover
- beetmover-l10n
- beetmover-repackage
- beetmover-repackage-l10n
only-for-attributes:
- nightly

1
taskcluster/ci/repackage-l10n/kind.yml
View File

@@ -7,7 +7,6 @@ loader: taskgraph.loader.single_dep:loader
transforms:
- taskgraph.transforms.repackage_l10n:transforms
- taskgraph.transforms.repackage:transforms
- taskgraph.transforms.job:transforms
- taskgraph.transforms.task:transforms
kind-dependencies:

1
taskcluster/ci/repackage/kind.yml
View File

@@ -6,7 +6,6 @@ loader: taskgraph.loader.single_dep:loader
transforms:
- taskgraph.transforms.repackage:transforms
- taskgraph.transforms.job:transforms
- taskgraph.transforms.task:transforms
kind-dependencies:

25
taskcluster/docs/kinds.rst
View File

@@ -4,7 +4,7 @@ Task Kinds
This section lists and documents the available task kinds.
build
-----
------
Builds are tasks that produce an installer or other output that can be run by
users or automated tests. This is more restrictive than most definitions of
@@ -12,11 +12,7 @@ users or automated tests. This is more restrictive than most definitions of
actions for static analysis or to produce instrumented artifacts.
build-signing
-------------
Many builds must be signed. The build-signing task takes the unsigned `build`
kind artifacts and passess them through signingscriptworker to a signing server
and returns signed results.
--------------
artifact-build
@@ -55,7 +51,7 @@ on an appropriate signing format. One signing job is created for each nightly-l1
job (usually chunked).
source-test
-----------
------------
Source-tests are tasks that run directly from the Gecko source. This can include linting,
unit tests, source-code analysis, or measurement work. While source-test tasks run from
@@ -157,7 +153,7 @@ The task definition used to create the image-building tasks is given in
Template <yaml-templates>`.
android-stuff
-------------
--------------
balrog
------
@@ -189,24 +185,15 @@ This separate kind uses logic specific to localized artifacts, such as including
the language in the final artifact names.
beetmover-repackage
-------------------
---------
Beetmover-repackage is beetmover but for tasks that need an intermediate step
between signing and packaging, such as OSX. For more details see the definitions
of the Beetmover kind above and the repackage kind below.
beetmover-repackage-l10n
------------------------
Same as beetmover-repackage but for l10n.
Beetmover L10n, takes specific artifacts, "Beets", and pushes them to a location outside
of Taskcluster's task artifacts, (archive.mozilla.org as one place) and in the
process determines the final location and a "pretty" name (versioned product name)
This separate kind uses logic specific to localized artifacts, such as including
the language in the final artifact names.
checksums-signing
-----------------
----------------
Checksums-signing take as input the checksums file generated by beetmover tasks
and sign it via the signing scriptworkers. Returns the same file signed and
additional detached signature.

17
taskcluster/taskgraph/transforms/beetmover_repackage.py
View File

@@ -155,19 +155,16 @@ def make_task_description(config, jobs):
dep_job.dependencies["docker-image"]
}
dependencies.update(docker_dependencies)
signing_name = "build-signing"
if job.get('locale'):
signing_name = "nightly-l10n-signing"
signing_dependencies = {signing_name:
dep_job.dependencies[signing_name]
signing_dependencies = {"build-signing":
dep_job.dependencies["build-signing"]
}
dependencies.update(signing_dependencies)
build_name = "build"
if job.get('locale'):
build_name = "unsigned-repack"
repackage_dependencies = {"repackage":
dep_job.label
}
dependencies.update(repackage_dependencies)
build_dependencies = {"build":
dep_job.dependencies[build_name]
dep_job.dependencies["build-signing"].replace("signing-", "build-")
}
dependencies.update(build_dependencies)

43
taskcluster/taskgraph/transforms/beetmover_repackage_l10n.py
View File

@@ -1,43 +0,0 @@
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
"""
Transform the signing task into an actual task description.
"""
from __future__ import absolute_import, print_function, unicode_literals
from taskgraph.transforms.base import TransformSequence
from taskgraph.util.treeherder import join_symbol
transforms = TransformSequence()
@transforms.add
def make_beetmover_description(config, jobs):
for job in jobs:
dep_job = job['dependent-task']
locale = dep_job.attributes.get('locale')
if not locale:
return
label = job.get('label',
"beetmover-{}-{}".format(locale, dep_job.label))
group = 'tc-BMR-L10n'
# add the locale code
symbol = locale
treeherder = {
'symbol': join_symbol(group, symbol),
}
beet_description = {
'dependent-task': dep_job,
'treeherder': treeherder,
'label': label,
'locale': locale,
}
yield beet_description

2
taskcluster/taskgraph/transforms/build_signing.py
View File

@@ -48,7 +48,7 @@ def make_signing_description(config, jobs):
job_specs = [
{
'artifacts': ['public/build/target.dmg'],
'format': 'macapp',
'format': 'dmg',
}, {
'artifacts': ['public/build/update/target.complete.mar'],
'format': 'mar',

4
taskcluster/taskgraph/transforms/job/__init__.py
View File

@@ -95,7 +95,7 @@ transforms = TransformSequence()
def validate(config, jobs):
for job in jobs:
yield validate_schema(job_description_schema, job,
"In job {!r}:".format(job.get('name', job.get('label'))))
"In job {!r}:".format(job['name']))
@transforms.add
@@ -133,7 +133,7 @@ def make_task_description(config, jobs):
if 'name' not in job:
raise Exception("job has neither a name nor a label")
job['label'] = '{}-{}'.format(config.kind, job['name'])
if job.get('name'):
if job['name']:
del job['name']
impl, os = worker_type_implementation(job['worker-type'])

11
taskcluster/taskgraph/transforms/job/common.py
View File

@@ -12,12 +12,9 @@ from __future__ import absolute_import, print_function, unicode_literals
SECRET_SCOPE = 'secrets:get:project/releng/gecko/{}/level-{}/{}'
def docker_worker_add_workspace_cache(config, job, taskdesc, extra=None):
def docker_worker_add_workspace_cache(config, job, taskdesc):
"""Add the workspace cache based on the build platform/type and level,
except on try where workspace caches are not used.
extra, is an optional kwarg passed in that supports extending the cache
key name to avoid undesired conflicts with other caches."""
except on try where workspace caches are not used."""
if config.params['project'] == 'try':
return
@@ -30,10 +27,6 @@ def docker_worker_add_workspace_cache(config, job, taskdesc, extra=None):
),
'mount-point': "/home/worker/workspace",
})
if extra:
taskdesc['worker']['caches'][-1]['name'] += '-{}'.format(
extra
)
def docker_worker_add_tc_vcs_cache(config, job, taskdesc):

7
taskcluster/taskgraph/transforms/job/mozharness.py
View File

@@ -49,10 +49,6 @@ mozharness_run_schema = Schema({
# --custom-build-variant-cfg value (not supported on Windows)
Optional('custom-build-variant-cfg'): basestring,
# Extra metadata to use toward the workspace caching.
# Only supported on docker-worker
Optional('extra-workspace-cache-key'): basestring,
# If not false, tooltool downloads will be enabled via relengAPIProxy
# for either just public files, or all files. Not supported on Windows
Required('tooltool-downloads', default=False): Any(
@@ -98,8 +94,7 @@ def mozharness_on_docker_worker_setup(config, job, taskdesc):
worker['taskcluster-proxy'] = run.get('taskcluster-proxy')
docker_worker_add_public_artifacts(config, job, taskdesc)
docker_worker_add_workspace_cache(config, job, taskdesc,
extra=run.get('extra-workspace-cache-key'))
docker_worker_add_workspace_cache(config, job, taskdesc)
support_vcs_checkout(config, job, taskdesc)
env = worker.setdefault('env', {})

2
taskcluster/taskgraph/transforms/nightly_l10n_signing.py
View File

@@ -33,7 +33,7 @@ def make_signing_description(config, jobs):
job_specs = [
{
'artifacts': ['public/build/{locale}/target.dmg'],
'format': 'macapp',
'format': 'dmg',
}, {
'artifacts': ['public/build/{locale}/target.complete.mar'],
'format': 'mar',

127
taskcluster/taskgraph/transforms/repackage.py
View File

@@ -67,18 +67,16 @@ def make_repackage_description(config, jobs):
label = job.get('label',
dep_job.label.replace("signing-", "repackage-"))
job['label'] = label
cot = job.setdefault('extra', {}).setdefault('chainOfTrust', {})
cot.setdefault('inputs', {})['docker-image'] = {"task-reference": "<docker-image>"}
yield job
@transforms.add
def make_job_description(config, jobs):
def make_task_description(config, jobs):
for job in jobs:
dep_job = job['dependent-task']
dependencies = {dep_job.attributes.get('kind'): dep_job.label}
if len(dep_job.dependencies) > 1:
raise NotImplementedError(
"Can't repackage a signing task with multiple dependencies")
treeherder = job.get('treeherder', {})
treeherder.setdefault('symbol', 'tc(Nr)')
@@ -87,79 +85,88 @@ def make_job_description(config, jobs):
treeherder.setdefault('platform', "{}/opt".format(dep_th_platform))
treeherder.setdefault('tier', 1)
treeherder.setdefault('kind', 'build')
signing_task = None
for dependency in dependencies.keys():
if 'signing' in dependency:
signing_task = dependency
signing_task_ref = "<{}>".format(signing_task)
attributes = {
'nightly': dep_job.attributes.get('nightly', False),
'build_platform': dep_job.attributes.get('build_platform'),
'build_type': dep_job.attributes.get('build_type'),
}
if job.get('locale'):
attributes['locale'] = job['locale']
command = ['/home/worker/bin/run-task',
# Various caches/volumes are default owned by root:root.
'--chown-recursive', '/home/worker/workspace',
'--chown-recursive', '/home/worker/tooltool-cache',
'--vcs-checkout', '/home/worker/workspace/build/src',
'--tools-checkout', '/home/worker/workspace/build/tools',
'--',
'/home/worker/workspace/build/src/taskcluster/scripts/builder/repackage.sh'
]
dependencies = {dep_job.attributes.get('kind'): dep_job.label}
if job.get('locale'):
input_string = 'https://queue.taskcluster.net/v1/task/<nightly-l10n-signing>/' + \
'artifacts/public/build/{}/target.tar.gz'
input_string = input_string.format(job['locale'])
else:
input_string = 'https://queue.taskcluster.net/v1/task/<build-signing>/artifacts/' + \
'public/build/target.tar.gz'
signed_input = {'task-reference': input_string}
level = config.params['level']
task_env = {}
locale_output_path = ""
if attributes['build_platform'].startswith('macosx'):
if job.get('locale'):
input_string = 'https://queue.taskcluster.net/v1/task/' + \
'{}/artifacts/public/build/{}/target.tar.gz'
input_string = input_string.format(signing_task_ref, job['locale'])
locale_output_path = "{}/".format(job['locale'])
else:
input_string = 'https://queue.taskcluster.net/v1/task/' + \
'{}/artifacts/public/build/target.tar.gz'.format(signing_task_ref)
task_env.update(
SIGNED_INPUT={'task-reference': input_string},
)
mozharness_config = ['repackage/osx_signed.py']
output_files = [{
'type': 'file',
'path': '/home/worker/workspace/build/artifacts/target.dmg',
'name': 'public/build/{}target.dmg'.format(locale_output_path),
}]
else:
raise Exception("Unexpected build platform for repackage")
run = {
'using': 'mozharness',
'script': 'mozharness/scripts/repackage.py',
'config': mozharness_config,
'job-script': 'taskcluster/scripts/builder/repackage.sh',
'actions': ['download_input', 'setup', 'repackage'],
'extra-workspace-cache-key': 'repackage',
}
if attributes["build_platform"].startswith('macosx'):
worker = {
'docker-image': {"in-tree": "desktop-build"},
'artifacts': output_files,
'env': task_env,
'chain-of-trust': True,
'max-run-time': 3600
}
run["tooltool-downloads"] = 'internal'
worker_type = 'aws-provisioner-v1/gecko-%s-b-macosx64' % level
cot = job.setdefault('extra', {}).setdefault('chainOfTrust', {})
cot.setdefault('inputs', {})['docker-image'] = {"task-reference": "<docker-image>"}
task = {
'label': job['label'],
'description': "{} Repackage".format(
dep_job.task["metadata"]["description"]),
'worker-type': worker_type,
'worker-type': 'aws-provisioner-v1/gecko-%s-b-macosx64' % level,
'dependencies': dependencies,
'attributes': attributes,
'run-on-projects': dep_job.attributes.get('run_on_projects'),
'treeherder': treeherder,
'routes': job.get('routes', []),
'extra': job.get('extra', {}),
'worker': worker,
'run': run,
'scopes':
['docker-worker:relengapi-proxy:tooltool.download.internal',
'secrets:get:project/taskcluster/gecko/hgfingerprint',
'docker-worker:relengapi-proxy:tooltool.download.public',
'project:releng:signing:format:dmg'],
'worker': {'implementation': 'docker-worker',
'os': 'linux',
'docker-image': {"in-tree": "desktop-build"},
'caches': [{
'type': 'persistent',
'name': 'tooltool-cache',
'mount-point': '/home/worker/tooltool-cache',
}, {
'type': 'persistent',
'name': 'level-%s-checkouts-v1' % level,
'mount-point': '/home/worker/checkouts',
}],
'artifacts': [{
'type': 'file',
'path': '/home/worker/workspace/build/upload/target.dmg',
'name': 'public/build/target.dmg',
}],
'env': {
'SIGNED_INPUT': signed_input,
'MOZ_BUILD_DATE': config.params['moz_build_date'],
'MH_BUILD_POOL': 'taskcluster',
'HG_STORE_PATH': '/home/worker/checkouts/hg-store',
'GECKO_HEAD_REV': config.params['head_rev'],
'MH_BRANCH': config.params['project'],
'MOZ_SCM_LEVEL': level,
'MOZHARNESS_ACTIONS': 'download_input setup repackage',
'NEED_XVFB': 'true',
'GECKO_BASE_REPOSITORY': config.params['base_repository'],
'TOOLTOOL_CACHE': '/home/worker/tooltool-cache',
'GECKO_HEAD_REPOSITORY': config.params['head_repository'],
'MOZHARNESS_CONFIG': 'repackage/osx_signed.py',
'USE_SCCACHE': '1',
'MOZHARNESS_SCRIPT': 'mozharness/scripts/repackage.py'
},
'command': command,
'chain-of-trust': True,
'relengapi-proxy': True,
'max-run-time': 3600
}
}
yield task

12
taskcluster/taskgraph/transforms/repackage_l10n.py
View File

@@ -17,17 +17,15 @@ def split_locales(config, jobs):
for job in jobs:
dep_job = job['dependent-task']
for locale in dep_job.attributes.get('chunk_locales', []):
job['locale'] = locale
label = dep_job.label.replace("signing-", "repackage-{}-".format(locale))
label = "repackage-{}-{}/{}".format(locale,
dep_job.attributes['build_platform'],
dep_job.attributes['build_type'],)
job['label'] = label
treeherder = job.get('treeherder', {})
treeherder['symbol'] = 'tc-L10n-Rpk({})'.format(locale)
yield {
'locale': locale,
'label': label,
'treeherder': treeherder,
'dependent-task': dep_job,
}
job['treeherder'] = treeherder
yield job

1
taskcluster/taskgraph/transforms/task.py
View File

@@ -451,7 +451,6 @@ GROUP_NAMES = {
'tc-L10n': 'Localised Repacks executed by Taskcluster',
'tc-L10n-Rpk': 'Localized Repackaged Repacks executed by Taskcluster',
'tc-BM-L10n': 'Beetmover for locales executed by Taskcluster',
'tc-BMR-L10n': 'Beetmover repackages for locales executed by Taskcluster',
'tc-Up': 'Balrog submission of updates, executed by Taskcluster',
'tc-cs': 'Checksum signing executed by Taskcluster',
'tc-BMcs': 'Beetmover checksums, executed by Taskcluster',

15
testing/mozharness/configs/repackage/osx_signed.py
View File

@@ -1,22 +1,15 @@
import os
config = {
"input_home": "{abs_work_dir}/inputs",
"input_filename": "target.tar.gz",
"output_filename": "target.dmg",
"input_home": "/home/worker/workspace/inputs",
"src_mozconfig": "browser/config/mozconfigs/macosx64/repack",
"download_config": {
"target.tar.gz": os.environ.get("SIGNED_INPUT"),
},
"repackage_config": [[
"dmg",
"-i", "{abs_work_dir}/inputs/target.tar.gz",
"-o", "{output_home}/target.dmg"
]],
# ToolTool
"tooltool_manifest_src": 'browser/config/tooltool-manifests/macosx64/cross-releng.manifest',
"tooltool_url": 'http://relengapi/tooltool/',
"tooltool_bootstrap": "setup.sh",
'tooltool_script': ["/builds/tooltool.py"],
'tooltool_cache': os.environ.get('TOOLTOOL_CACHE'),
}

84
testing/mozharness/scripts/repackage.py
View File

@@ -10,6 +10,21 @@ from mozharness.mozilla.mock import ERROR_MSGS
class Repackage(BaseScript):
config_options = [[
['--signed-input', ],
{"action": "store",
"dest": "signed_input",
"type": "string",
"default": os.environ.get('SIGNED_INPUT'),
"help": "Specify the signed input (url)"}
], [
['--output-file', ],
{"action": "store",
"dest": "output_file",
"type": "string",
"help": "Specify the output filename"}
]]
def __init__(self, require_config_file=False):
script_kwargs = {
'all_actions': [
@@ -20,22 +35,24 @@ class Repackage(BaseScript):
}
BaseScript.__init__(
self,
config_options=self.config_options,
require_config_file=require_config_file,
**script_kwargs
)
# Assert we have it either passed in or in environment
assert self.config.get('signed_input'), \
"Must pass --signed-input or be set in the environment as SIGNED_INPUT"
def download_input(self):
config = self.config
dirs = self.query_abs_dirs()
input_home = config['input_home'].format(**dirs)
for path, url in config["download_config"].items():
url = config['signed_input']
status = self.download_file(url=url,
file_name=path,
parent_dir=input_home)
file_name=config['input_filename'],
parent_dir=config['input_home'])
if not status:
self.fatal("Unable to fetch signed input from %s" % url)
self.fatal("Unable to fetch signed input from %s" % config['signed_input'])
def setup(self):
self._run_tooltool()
@@ -52,7 +69,6 @@ class Repackage(BaseScript):
dirs = {}
dirs['abs_tools_dir'] = os.path.join(abs_dirs['abs_work_dir'], 'tools')
dirs['abs_mozilla_dir'] = os.path.join(abs_dirs['abs_work_dir'], 'src')
dirs['output_home'] = os.path.join(abs_dirs['abs_work_dir'], 'artifacts')
for key in dirs.keys():
if key not in abs_dirs:
abs_dirs[key] = dirs[key]
@@ -62,15 +78,13 @@ class Repackage(BaseScript):
def repackage(self):
config = self.config
dirs = self.query_abs_dirs()
# Make sure the upload dir is around.
self.mkdir_p(dirs['output_home'])
for repack_config in config["repackage_config"]:
command = [sys.executable, 'mach', '--log-no-times', 'repackage'] + \
[arg.format(**dirs)
for arg in list(repack_config)]
self.run_command(
infile = os.path.join(config['input_home'], config['input_filename'])
outfile = os.path.join(dirs['abs_upload_dir'], config['output_filename'])
command = [sys.executable, 'mach', '--log-no-times', 'repackage',
'dmg',
'--input', infile,
'--output', outfile]
return self.run_command(
command=command,
cwd=dirs['abs_mozilla_dir'],
halt_on_failure=True,
@@ -84,46 +98,24 @@ class Repackage(BaseScript):
manifest_src = config.get('tooltool_manifest_src')
if not manifest_src:
return self.warning(ERROR_MSGS['tooltool_manifest_undetermined'])
fetch_script_path = os.path.join(dirs['abs_tools_dir'],
'scripts/tooltool/tooltool_wrapper.sh')
tooltool_manifest_path = os.path.join(dirs['abs_mozilla_dir'],
manifest_src)
cmd = [
sys.executable, '-u',
os.path.join(dirs['abs_mozilla_dir'], 'mach'),
'artifact',
'toolchain',
'-v',
'--retry', '4',
'--tooltool-manifest',
'sh',
fetch_script_path,
tooltool_manifest_path,
'--tooltool-url',
config['tooltool_url'],
config['tooltool_bootstrap'],
]
auth_file = self._get_tooltool_auth_file()
if auth_file:
cmd.extend(['--authentication-file', auth_file])
cmd.extend(config['tooltool_script'])
cache = config.get('tooltool_cache')
if cache:
cmd.extend(['--cache-dir', cache])
cmd.extend(['-c', cache])
self.info(str(cmd))
self.run_command(cmd, cwd=dirs['abs_mozilla_dir'], halt_on_failure=True)
def _get_tooltool_auth_file(self):
# set the default authentication file based on platform; this
# corresponds to where puppet puts the token
if 'tooltool_authentication_file' in self.config:
fn = self.config['tooltool_authentication_file']
elif self._is_windows():
fn = r'c:\builds\relengapi.tok'
else:
fn = '/builds/relengapi.tok'
# if the file doesn't exist, don't pass it to tooltool (it will just
# fail). In taskcluster, this will work OK as the relengapi-proxy will
# take care of auth. Everywhere else, we'll get auth failures if
# necessary.
if os.path.exists(fn):
return fn
def _get_mozconfig(self):
"""assign mozconfig."""
c = self.config