Backed out 5 changesets (bug 1373326) for breaking the decision task a=backout
Backed out changeset 7f0177a71077 (bug 1373326) Backed out changeset a678d4b0ee78 (bug 1373326) Backed out changeset 2fdc6c2ad384 (bug 1373326) Backed out changeset 97b785a31a3c (bug 1373326) Backed out changeset a5fcfe57e6d4 (bug 1373326) MozReview-Commit-ID: 9kcJzI1hFja
This commit is contained in:
@@ -12,7 +12,6 @@ kind-dependencies:
|
|||||||
- beetmover
|
- beetmover
|
||||||
- beetmover-l10n
|
- beetmover-l10n
|
||||||
- beetmover-repackage
|
- beetmover-repackage
|
||||||
- beetmover-repackage-l10n
|
|
||||||
|
|
||||||
only-for-attributes:
|
only-for-attributes:
|
||||||
- nightly
|
- nightly
|
||||||
|
|||||||
@@ -14,6 +14,3 @@ kind-dependencies:
|
|||||||
|
|
||||||
only-for-attributes:
|
only-for-attributes:
|
||||||
- nightly
|
- nightly
|
||||||
|
|
||||||
not-for-build-platforms:
|
|
||||||
- macosx64-nightly/opt
|
|
||||||
|
|||||||
@@ -1,16 +0,0 @@
|
|||||||
# This Source Code Form is subject to the terms of the Mozilla Public
|
|
||||||
# License, v. 2.0. If a copy of the MPL was not distributed with this
|
|
||||||
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
|
||||||
|
|
||||||
loader: taskgraph.loader.single_dep:loader
|
|
||||||
|
|
||||||
transforms:
|
|
||||||
- taskgraph.transforms.beetmover_repackage_l10n:transforms
|
|
||||||
- taskgraph.transforms.beetmover_repackage:transforms
|
|
||||||
- taskgraph.transforms.task:transforms
|
|
||||||
|
|
||||||
kind-dependencies:
|
|
||||||
- repackage-l10n
|
|
||||||
|
|
||||||
only-for-build-platforms:
|
|
||||||
- macosx64-nightly/opt
|
|
||||||
@@ -12,7 +12,6 @@ kind-dependencies:
|
|||||||
- beetmover
|
- beetmover
|
||||||
- beetmover-l10n
|
- beetmover-l10n
|
||||||
- beetmover-repackage
|
- beetmover-repackage
|
||||||
- beetmover-repackage-l10n
|
|
||||||
|
|
||||||
only-for-attributes:
|
only-for-attributes:
|
||||||
- nightly
|
- nightly
|
||||||
|
|||||||
@@ -7,7 +7,6 @@ loader: taskgraph.loader.single_dep:loader
|
|||||||
transforms:
|
transforms:
|
||||||
- taskgraph.transforms.repackage_l10n:transforms
|
- taskgraph.transforms.repackage_l10n:transforms
|
||||||
- taskgraph.transforms.repackage:transforms
|
- taskgraph.transforms.repackage:transforms
|
||||||
- taskgraph.transforms.job:transforms
|
|
||||||
- taskgraph.transforms.task:transforms
|
- taskgraph.transforms.task:transforms
|
||||||
|
|
||||||
kind-dependencies:
|
kind-dependencies:
|
||||||
|
|||||||
@@ -6,7 +6,6 @@ loader: taskgraph.loader.single_dep:loader
|
|||||||
|
|
||||||
transforms:
|
transforms:
|
||||||
- taskgraph.transforms.repackage:transforms
|
- taskgraph.transforms.repackage:transforms
|
||||||
- taskgraph.transforms.job:transforms
|
|
||||||
- taskgraph.transforms.task:transforms
|
- taskgraph.transforms.task:transforms
|
||||||
|
|
||||||
kind-dependencies:
|
kind-dependencies:
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ Task Kinds
|
|||||||
This section lists and documents the available task kinds.
|
This section lists and documents the available task kinds.
|
||||||
|
|
||||||
build
|
build
|
||||||
-----
|
------
|
||||||
|
|
||||||
Builds are tasks that produce an installer or other output that can be run by
|
Builds are tasks that produce an installer or other output that can be run by
|
||||||
users or automated tests. This is more restrictive than most definitions of
|
users or automated tests. This is more restrictive than most definitions of
|
||||||
@@ -12,11 +12,7 @@ users or automated tests. This is more restrictive than most definitions of
|
|||||||
actions for static analysis or to produce instrumented artifacts.
|
actions for static analysis or to produce instrumented artifacts.
|
||||||
|
|
||||||
build-signing
|
build-signing
|
||||||
-------------
|
--------------
|
||||||
|
|
||||||
Many builds must be signed. The build-signing task takes the unsigned `build`
|
|
||||||
kind artifacts and passess them through signingscriptworker to a signing server
|
|
||||||
and returns signed results.
|
|
||||||
|
|
||||||
|
|
||||||
artifact-build
|
artifact-build
|
||||||
@@ -55,7 +51,7 @@ on an appropriate signing format. One signing job is created for each nightly-l1
|
|||||||
job (usually chunked).
|
job (usually chunked).
|
||||||
|
|
||||||
source-test
|
source-test
|
||||||
-----------
|
------------
|
||||||
|
|
||||||
Source-tests are tasks that run directly from the Gecko source. This can include linting,
|
Source-tests are tasks that run directly from the Gecko source. This can include linting,
|
||||||
unit tests, source-code analysis, or measurement work. While source-test tasks run from
|
unit tests, source-code analysis, or measurement work. While source-test tasks run from
|
||||||
@@ -157,7 +153,7 @@ The task definition used to create the image-building tasks is given in
|
|||||||
Template <yaml-templates>`.
|
Template <yaml-templates>`.
|
||||||
|
|
||||||
android-stuff
|
android-stuff
|
||||||
-------------
|
--------------
|
||||||
|
|
||||||
balrog
|
balrog
|
||||||
------
|
------
|
||||||
@@ -189,24 +185,15 @@ This separate kind uses logic specific to localized artifacts, such as including
|
|||||||
the language in the final artifact names.
|
the language in the final artifact names.
|
||||||
|
|
||||||
beetmover-repackage
|
beetmover-repackage
|
||||||
-------------------
|
---------
|
||||||
|
|
||||||
Beetmover-repackage is beetmover but for tasks that need an intermediate step
|
Beetmover-repackage is beetmover but for tasks that need an intermediate step
|
||||||
between signing and packaging, such as OSX. For more details see the definitions
|
between signing and packaging, such as OSX. For more details see the definitions
|
||||||
of the Beetmover kind above and the repackage kind below.
|
of the Beetmover kind above and the repackage kind below.
|
||||||
|
|
||||||
beetmover-repackage-l10n
|
|
||||||
------------------------
|
|
||||||
Same as beetmover-repackage but for l10n.
|
|
||||||
|
|
||||||
Beetmover L10n, takes specific artifacts, "Beets", and pushes them to a location outside
|
|
||||||
of Taskcluster's task artifacts, (archive.mozilla.org as one place) and in the
|
|
||||||
process determines the final location and a "pretty" name (versioned product name)
|
|
||||||
This separate kind uses logic specific to localized artifacts, such as including
|
|
||||||
the language in the final artifact names.
|
|
||||||
|
|
||||||
checksums-signing
|
checksums-signing
|
||||||
-----------------
|
----------------
|
||||||
Checksums-signing take as input the checksums file generated by beetmover tasks
|
Checksums-signing take as input the checksums file generated by beetmover tasks
|
||||||
and sign it via the signing scriptworkers. Returns the same file signed and
|
and sign it via the signing scriptworkers. Returns the same file signed and
|
||||||
additional detached signature.
|
additional detached signature.
|
||||||
|
|||||||
@@ -155,19 +155,16 @@ def make_task_description(config, jobs):
|
|||||||
dep_job.dependencies["docker-image"]
|
dep_job.dependencies["docker-image"]
|
||||||
}
|
}
|
||||||
dependencies.update(docker_dependencies)
|
dependencies.update(docker_dependencies)
|
||||||
signing_name = "build-signing"
|
signing_dependencies = {"build-signing":
|
||||||
if job.get('locale'):
|
dep_job.dependencies["build-signing"]
|
||||||
signing_name = "nightly-l10n-signing"
|
|
||||||
signing_dependencies = {signing_name:
|
|
||||||
dep_job.dependencies[signing_name]
|
|
||||||
}
|
}
|
||||||
dependencies.update(signing_dependencies)
|
dependencies.update(signing_dependencies)
|
||||||
|
repackage_dependencies = {"repackage":
|
||||||
build_name = "build"
|
dep_job.label
|
||||||
if job.get('locale'):
|
}
|
||||||
build_name = "unsigned-repack"
|
dependencies.update(repackage_dependencies)
|
||||||
build_dependencies = {"build":
|
build_dependencies = {"build":
|
||||||
dep_job.dependencies[build_name]
|
dep_job.dependencies["build-signing"].replace("signing-", "build-")
|
||||||
}
|
}
|
||||||
dependencies.update(build_dependencies)
|
dependencies.update(build_dependencies)
|
||||||
|
|
||||||
|
|||||||
@@ -1,43 +0,0 @@
|
|||||||
# This Source Code Form is subject to the terms of the Mozilla Public
|
|
||||||
# License, v. 2.0. If a copy of the MPL was not distributed with this
|
|
||||||
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
|
||||||
"""
|
|
||||||
Transform the signing task into an actual task description.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import absolute_import, print_function, unicode_literals
|
|
||||||
|
|
||||||
from taskgraph.transforms.base import TransformSequence
|
|
||||||
from taskgraph.util.treeherder import join_symbol
|
|
||||||
|
|
||||||
transforms = TransformSequence()
|
|
||||||
|
|
||||||
|
|
||||||
@transforms.add
|
|
||||||
def make_beetmover_description(config, jobs):
|
|
||||||
for job in jobs:
|
|
||||||
dep_job = job['dependent-task']
|
|
||||||
|
|
||||||
locale = dep_job.attributes.get('locale')
|
|
||||||
if not locale:
|
|
||||||
return
|
|
||||||
|
|
||||||
label = job.get('label',
|
|
||||||
"beetmover-{}-{}".format(locale, dep_job.label))
|
|
||||||
|
|
||||||
group = 'tc-BMR-L10n'
|
|
||||||
|
|
||||||
# add the locale code
|
|
||||||
symbol = locale
|
|
||||||
|
|
||||||
treeherder = {
|
|
||||||
'symbol': join_symbol(group, symbol),
|
|
||||||
}
|
|
||||||
|
|
||||||
beet_description = {
|
|
||||||
'dependent-task': dep_job,
|
|
||||||
'treeherder': treeherder,
|
|
||||||
'label': label,
|
|
||||||
'locale': locale,
|
|
||||||
}
|
|
||||||
yield beet_description
|
|
||||||
@@ -48,7 +48,7 @@ def make_signing_description(config, jobs):
|
|||||||
job_specs = [
|
job_specs = [
|
||||||
{
|
{
|
||||||
'artifacts': ['public/build/target.dmg'],
|
'artifacts': ['public/build/target.dmg'],
|
||||||
'format': 'macapp',
|
'format': 'dmg',
|
||||||
}, {
|
}, {
|
||||||
'artifacts': ['public/build/update/target.complete.mar'],
|
'artifacts': ['public/build/update/target.complete.mar'],
|
||||||
'format': 'mar',
|
'format': 'mar',
|
||||||
|
|||||||
@@ -95,7 +95,7 @@ transforms = TransformSequence()
|
|||||||
def validate(config, jobs):
|
def validate(config, jobs):
|
||||||
for job in jobs:
|
for job in jobs:
|
||||||
yield validate_schema(job_description_schema, job,
|
yield validate_schema(job_description_schema, job,
|
||||||
"In job {!r}:".format(job.get('name', job.get('label'))))
|
"In job {!r}:".format(job['name']))
|
||||||
|
|
||||||
|
|
||||||
@transforms.add
|
@transforms.add
|
||||||
@@ -133,7 +133,7 @@ def make_task_description(config, jobs):
|
|||||||
if 'name' not in job:
|
if 'name' not in job:
|
||||||
raise Exception("job has neither a name nor a label")
|
raise Exception("job has neither a name nor a label")
|
||||||
job['label'] = '{}-{}'.format(config.kind, job['name'])
|
job['label'] = '{}-{}'.format(config.kind, job['name'])
|
||||||
if job.get('name'):
|
if job['name']:
|
||||||
del job['name']
|
del job['name']
|
||||||
|
|
||||||
impl, os = worker_type_implementation(job['worker-type'])
|
impl, os = worker_type_implementation(job['worker-type'])
|
||||||
|
|||||||
@@ -12,12 +12,9 @@ from __future__ import absolute_import, print_function, unicode_literals
|
|||||||
SECRET_SCOPE = 'secrets:get:project/releng/gecko/{}/level-{}/{}'
|
SECRET_SCOPE = 'secrets:get:project/releng/gecko/{}/level-{}/{}'
|
||||||
|
|
||||||
|
|
||||||
def docker_worker_add_workspace_cache(config, job, taskdesc, extra=None):
|
def docker_worker_add_workspace_cache(config, job, taskdesc):
|
||||||
"""Add the workspace cache based on the build platform/type and level,
|
"""Add the workspace cache based on the build platform/type and level,
|
||||||
except on try where workspace caches are not used.
|
except on try where workspace caches are not used."""
|
||||||
|
|
||||||
extra, is an optional kwarg passed in that supports extending the cache
|
|
||||||
key name to avoid undesired conflicts with other caches."""
|
|
||||||
if config.params['project'] == 'try':
|
if config.params['project'] == 'try':
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -30,10 +27,6 @@ def docker_worker_add_workspace_cache(config, job, taskdesc, extra=None):
|
|||||||
),
|
),
|
||||||
'mount-point': "/home/worker/workspace",
|
'mount-point': "/home/worker/workspace",
|
||||||
})
|
})
|
||||||
if extra:
|
|
||||||
taskdesc['worker']['caches'][-1]['name'] += '-{}'.format(
|
|
||||||
extra
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def docker_worker_add_tc_vcs_cache(config, job, taskdesc):
|
def docker_worker_add_tc_vcs_cache(config, job, taskdesc):
|
||||||
|
|||||||
@@ -49,10 +49,6 @@ mozharness_run_schema = Schema({
|
|||||||
# --custom-build-variant-cfg value (not supported on Windows)
|
# --custom-build-variant-cfg value (not supported on Windows)
|
||||||
Optional('custom-build-variant-cfg'): basestring,
|
Optional('custom-build-variant-cfg'): basestring,
|
||||||
|
|
||||||
# Extra metadata to use toward the workspace caching.
|
|
||||||
# Only supported on docker-worker
|
|
||||||
Optional('extra-workspace-cache-key'): basestring,
|
|
||||||
|
|
||||||
# If not false, tooltool downloads will be enabled via relengAPIProxy
|
# If not false, tooltool downloads will be enabled via relengAPIProxy
|
||||||
# for either just public files, or all files. Not supported on Windows
|
# for either just public files, or all files. Not supported on Windows
|
||||||
Required('tooltool-downloads', default=False): Any(
|
Required('tooltool-downloads', default=False): Any(
|
||||||
@@ -98,8 +94,7 @@ def mozharness_on_docker_worker_setup(config, job, taskdesc):
|
|||||||
worker['taskcluster-proxy'] = run.get('taskcluster-proxy')
|
worker['taskcluster-proxy'] = run.get('taskcluster-proxy')
|
||||||
|
|
||||||
docker_worker_add_public_artifacts(config, job, taskdesc)
|
docker_worker_add_public_artifacts(config, job, taskdesc)
|
||||||
docker_worker_add_workspace_cache(config, job, taskdesc,
|
docker_worker_add_workspace_cache(config, job, taskdesc)
|
||||||
extra=run.get('extra-workspace-cache-key'))
|
|
||||||
support_vcs_checkout(config, job, taskdesc)
|
support_vcs_checkout(config, job, taskdesc)
|
||||||
|
|
||||||
env = worker.setdefault('env', {})
|
env = worker.setdefault('env', {})
|
||||||
|
|||||||
@@ -33,7 +33,7 @@ def make_signing_description(config, jobs):
|
|||||||
job_specs = [
|
job_specs = [
|
||||||
{
|
{
|
||||||
'artifacts': ['public/build/{locale}/target.dmg'],
|
'artifacts': ['public/build/{locale}/target.dmg'],
|
||||||
'format': 'macapp',
|
'format': 'dmg',
|
||||||
}, {
|
}, {
|
||||||
'artifacts': ['public/build/{locale}/target.complete.mar'],
|
'artifacts': ['public/build/{locale}/target.complete.mar'],
|
||||||
'format': 'mar',
|
'format': 'mar',
|
||||||
|
|||||||
@@ -67,18 +67,16 @@ def make_repackage_description(config, jobs):
|
|||||||
label = job.get('label',
|
label = job.get('label',
|
||||||
dep_job.label.replace("signing-", "repackage-"))
|
dep_job.label.replace("signing-", "repackage-"))
|
||||||
job['label'] = label
|
job['label'] = label
|
||||||
|
cot = job.setdefault('extra', {}).setdefault('chainOfTrust', {})
|
||||||
|
cot.setdefault('inputs', {})['docker-image'] = {"task-reference": "<docker-image>"}
|
||||||
|
|
||||||
yield job
|
yield job
|
||||||
|
|
||||||
|
|
||||||
@transforms.add
|
@transforms.add
|
||||||
def make_job_description(config, jobs):
|
def make_task_description(config, jobs):
|
||||||
for job in jobs:
|
for job in jobs:
|
||||||
dep_job = job['dependent-task']
|
dep_job = job['dependent-task']
|
||||||
dependencies = {dep_job.attributes.get('kind'): dep_job.label}
|
|
||||||
if len(dep_job.dependencies) > 1:
|
|
||||||
raise NotImplementedError(
|
|
||||||
"Can't repackage a signing task with multiple dependencies")
|
|
||||||
|
|
||||||
treeherder = job.get('treeherder', {})
|
treeherder = job.get('treeherder', {})
|
||||||
treeherder.setdefault('symbol', 'tc(Nr)')
|
treeherder.setdefault('symbol', 'tc(Nr)')
|
||||||
@@ -87,79 +85,88 @@ def make_job_description(config, jobs):
|
|||||||
treeherder.setdefault('platform', "{}/opt".format(dep_th_platform))
|
treeherder.setdefault('platform', "{}/opt".format(dep_th_platform))
|
||||||
treeherder.setdefault('tier', 1)
|
treeherder.setdefault('tier', 1)
|
||||||
treeherder.setdefault('kind', 'build')
|
treeherder.setdefault('kind', 'build')
|
||||||
signing_task = None
|
|
||||||
for dependency in dependencies.keys():
|
|
||||||
if 'signing' in dependency:
|
|
||||||
signing_task = dependency
|
|
||||||
signing_task_ref = "<{}>".format(signing_task)
|
|
||||||
|
|
||||||
attributes = {
|
attributes = {
|
||||||
'nightly': dep_job.attributes.get('nightly', False),
|
'nightly': dep_job.attributes.get('nightly', False),
|
||||||
'build_platform': dep_job.attributes.get('build_platform'),
|
'build_platform': dep_job.attributes.get('build_platform'),
|
||||||
'build_type': dep_job.attributes.get('build_type'),
|
'build_type': dep_job.attributes.get('build_type'),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
command = ['/home/worker/bin/run-task',
|
||||||
|
# Various caches/volumes are default owned by root:root.
|
||||||
|
'--chown-recursive', '/home/worker/workspace',
|
||||||
|
'--chown-recursive', '/home/worker/tooltool-cache',
|
||||||
|
'--vcs-checkout', '/home/worker/workspace/build/src',
|
||||||
|
'--tools-checkout', '/home/worker/workspace/build/tools',
|
||||||
|
'--',
|
||||||
|
'/home/worker/workspace/build/src/taskcluster/scripts/builder/repackage.sh'
|
||||||
|
]
|
||||||
|
|
||||||
|
dependencies = {dep_job.attributes.get('kind'): dep_job.label}
|
||||||
if job.get('locale'):
|
if job.get('locale'):
|
||||||
attributes['locale'] = job['locale']
|
input_string = 'https://queue.taskcluster.net/v1/task/<nightly-l10n-signing>/' + \
|
||||||
|
'artifacts/public/build/{}/target.tar.gz'
|
||||||
level = config.params['level']
|
input_string = input_string.format(job['locale'])
|
||||||
|
|
||||||
task_env = {}
|
|
||||||
locale_output_path = ""
|
|
||||||
if attributes['build_platform'].startswith('macosx'):
|
|
||||||
if job.get('locale'):
|
|
||||||
input_string = 'https://queue.taskcluster.net/v1/task/' + \
|
|
||||||
'{}/artifacts/public/build/{}/target.tar.gz'
|
|
||||||
input_string = input_string.format(signing_task_ref, job['locale'])
|
|
||||||
locale_output_path = "{}/".format(job['locale'])
|
|
||||||
else:
|
|
||||||
input_string = 'https://queue.taskcluster.net/v1/task/' + \
|
|
||||||
'{}/artifacts/public/build/target.tar.gz'.format(signing_task_ref)
|
|
||||||
task_env.update(
|
|
||||||
SIGNED_INPUT={'task-reference': input_string},
|
|
||||||
)
|
|
||||||
mozharness_config = ['repackage/osx_signed.py']
|
|
||||||
output_files = [{
|
|
||||||
'type': 'file',
|
|
||||||
'path': '/home/worker/workspace/build/artifacts/target.dmg',
|
|
||||||
'name': 'public/build/{}target.dmg'.format(locale_output_path),
|
|
||||||
}]
|
|
||||||
else:
|
else:
|
||||||
raise Exception("Unexpected build platform for repackage")
|
input_string = 'https://queue.taskcluster.net/v1/task/<build-signing>/artifacts/' + \
|
||||||
|
'public/build/target.tar.gz'
|
||||||
run = {
|
signed_input = {'task-reference': input_string}
|
||||||
'using': 'mozharness',
|
level = config.params['level']
|
||||||
'script': 'mozharness/scripts/repackage.py',
|
|
||||||
'config': mozharness_config,
|
|
||||||
'job-script': 'taskcluster/scripts/builder/repackage.sh',
|
|
||||||
'actions': ['download_input', 'setup', 'repackage'],
|
|
||||||
'extra-workspace-cache-key': 'repackage',
|
|
||||||
}
|
|
||||||
|
|
||||||
if attributes["build_platform"].startswith('macosx'):
|
|
||||||
worker = {
|
|
||||||
'docker-image': {"in-tree": "desktop-build"},
|
|
||||||
'artifacts': output_files,
|
|
||||||
'env': task_env,
|
|
||||||
'chain-of-trust': True,
|
|
||||||
'max-run-time': 3600
|
|
||||||
}
|
|
||||||
run["tooltool-downloads"] = 'internal'
|
|
||||||
worker_type = 'aws-provisioner-v1/gecko-%s-b-macosx64' % level
|
|
||||||
cot = job.setdefault('extra', {}).setdefault('chainOfTrust', {})
|
|
||||||
cot.setdefault('inputs', {})['docker-image'] = {"task-reference": "<docker-image>"}
|
|
||||||
|
|
||||||
task = {
|
task = {
|
||||||
'label': job['label'],
|
'label': job['label'],
|
||||||
'description': "{} Repackage".format(
|
'description': "{} Repackage".format(
|
||||||
dep_job.task["metadata"]["description"]),
|
dep_job.task["metadata"]["description"]),
|
||||||
'worker-type': worker_type,
|
'worker-type': 'aws-provisioner-v1/gecko-%s-b-macosx64' % level,
|
||||||
'dependencies': dependencies,
|
'dependencies': dependencies,
|
||||||
'attributes': attributes,
|
'attributes': attributes,
|
||||||
'run-on-projects': dep_job.attributes.get('run_on_projects'),
|
'run-on-projects': dep_job.attributes.get('run_on_projects'),
|
||||||
'treeherder': treeherder,
|
'treeherder': treeherder,
|
||||||
'routes': job.get('routes', []),
|
'routes': job.get('routes', []),
|
||||||
'extra': job.get('extra', {}),
|
'extra': job.get('extra', {}),
|
||||||
'worker': worker,
|
'scopes':
|
||||||
'run': run,
|
['docker-worker:relengapi-proxy:tooltool.download.internal',
|
||||||
|
'secrets:get:project/taskcluster/gecko/hgfingerprint',
|
||||||
|
'docker-worker:relengapi-proxy:tooltool.download.public',
|
||||||
|
'project:releng:signing:format:dmg'],
|
||||||
|
'worker': {'implementation': 'docker-worker',
|
||||||
|
'os': 'linux',
|
||||||
|
'docker-image': {"in-tree": "desktop-build"},
|
||||||
|
'caches': [{
|
||||||
|
'type': 'persistent',
|
||||||
|
'name': 'tooltool-cache',
|
||||||
|
'mount-point': '/home/worker/tooltool-cache',
|
||||||
|
}, {
|
||||||
|
'type': 'persistent',
|
||||||
|
'name': 'level-%s-checkouts-v1' % level,
|
||||||
|
'mount-point': '/home/worker/checkouts',
|
||||||
|
}],
|
||||||
|
'artifacts': [{
|
||||||
|
'type': 'file',
|
||||||
|
'path': '/home/worker/workspace/build/upload/target.dmg',
|
||||||
|
'name': 'public/build/target.dmg',
|
||||||
|
}],
|
||||||
|
'env': {
|
||||||
|
'SIGNED_INPUT': signed_input,
|
||||||
|
'MOZ_BUILD_DATE': config.params['moz_build_date'],
|
||||||
|
'MH_BUILD_POOL': 'taskcluster',
|
||||||
|
'HG_STORE_PATH': '/home/worker/checkouts/hg-store',
|
||||||
|
'GECKO_HEAD_REV': config.params['head_rev'],
|
||||||
|
'MH_BRANCH': config.params['project'],
|
||||||
|
'MOZ_SCM_LEVEL': level,
|
||||||
|
'MOZHARNESS_ACTIONS': 'download_input setup repackage',
|
||||||
|
'NEED_XVFB': 'true',
|
||||||
|
'GECKO_BASE_REPOSITORY': config.params['base_repository'],
|
||||||
|
'TOOLTOOL_CACHE': '/home/worker/tooltool-cache',
|
||||||
|
'GECKO_HEAD_REPOSITORY': config.params['head_repository'],
|
||||||
|
'MOZHARNESS_CONFIG': 'repackage/osx_signed.py',
|
||||||
|
'USE_SCCACHE': '1',
|
||||||
|
'MOZHARNESS_SCRIPT': 'mozharness/scripts/repackage.py'
|
||||||
|
},
|
||||||
|
'command': command,
|
||||||
|
'chain-of-trust': True,
|
||||||
|
'relengapi-proxy': True,
|
||||||
|
'max-run-time': 3600
|
||||||
|
}
|
||||||
}
|
}
|
||||||
yield task
|
yield task
|
||||||
|
|||||||
@@ -17,17 +17,15 @@ def split_locales(config, jobs):
|
|||||||
for job in jobs:
|
for job in jobs:
|
||||||
dep_job = job['dependent-task']
|
dep_job = job['dependent-task']
|
||||||
for locale in dep_job.attributes.get('chunk_locales', []):
|
for locale in dep_job.attributes.get('chunk_locales', []):
|
||||||
|
job['locale'] = locale
|
||||||
|
|
||||||
label = dep_job.label.replace("signing-", "repackage-{}-".format(locale))
|
label = dep_job.label.replace("signing-", "repackage-{}-".format(locale))
|
||||||
label = "repackage-{}-{}/{}".format(locale,
|
label = "repackage-{}-{}/{}".format(locale,
|
||||||
dep_job.attributes['build_platform'],
|
dep_job.attributes['build_platform'],
|
||||||
dep_job.attributes['build_type'],)
|
dep_job.attributes['build_type'],)
|
||||||
|
job['label'] = label
|
||||||
|
|
||||||
treeherder = job.get('treeherder', {})
|
treeherder = job.get('treeherder', {})
|
||||||
treeherder['symbol'] = 'tc-L10n-Rpk({})'.format(locale)
|
treeherder['symbol'] = 'tc-L10n-Rpk({})'.format(locale)
|
||||||
|
job['treeherder'] = treeherder
|
||||||
yield {
|
yield job
|
||||||
'locale': locale,
|
|
||||||
'label': label,
|
|
||||||
'treeherder': treeherder,
|
|
||||||
'dependent-task': dep_job,
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -451,7 +451,6 @@ GROUP_NAMES = {
|
|||||||
'tc-L10n': 'Localised Repacks executed by Taskcluster',
|
'tc-L10n': 'Localised Repacks executed by Taskcluster',
|
||||||
'tc-L10n-Rpk': 'Localized Repackaged Repacks executed by Taskcluster',
|
'tc-L10n-Rpk': 'Localized Repackaged Repacks executed by Taskcluster',
|
||||||
'tc-BM-L10n': 'Beetmover for locales executed by Taskcluster',
|
'tc-BM-L10n': 'Beetmover for locales executed by Taskcluster',
|
||||||
'tc-BMR-L10n': 'Beetmover repackages for locales executed by Taskcluster',
|
|
||||||
'tc-Up': 'Balrog submission of updates, executed by Taskcluster',
|
'tc-Up': 'Balrog submission of updates, executed by Taskcluster',
|
||||||
'tc-cs': 'Checksum signing executed by Taskcluster',
|
'tc-cs': 'Checksum signing executed by Taskcluster',
|
||||||
'tc-BMcs': 'Beetmover checksums, executed by Taskcluster',
|
'tc-BMcs': 'Beetmover checksums, executed by Taskcluster',
|
||||||
|
|||||||
@@ -1,22 +1,15 @@
|
|||||||
import os
|
import os
|
||||||
|
|
||||||
config = {
|
config = {
|
||||||
"input_home": "{abs_work_dir}/inputs",
|
"input_filename": "target.tar.gz",
|
||||||
|
"output_filename": "target.dmg",
|
||||||
|
"input_home": "/home/worker/workspace/inputs",
|
||||||
"src_mozconfig": "browser/config/mozconfigs/macosx64/repack",
|
"src_mozconfig": "browser/config/mozconfigs/macosx64/repack",
|
||||||
|
|
||||||
"download_config": {
|
|
||||||
"target.tar.gz": os.environ.get("SIGNED_INPUT"),
|
|
||||||
},
|
|
||||||
|
|
||||||
"repackage_config": [[
|
|
||||||
"dmg",
|
|
||||||
"-i", "{abs_work_dir}/inputs/target.tar.gz",
|
|
||||||
"-o", "{output_home}/target.dmg"
|
|
||||||
]],
|
|
||||||
|
|
||||||
# ToolTool
|
# ToolTool
|
||||||
"tooltool_manifest_src": 'browser/config/tooltool-manifests/macosx64/cross-releng.manifest',
|
"tooltool_manifest_src": 'browser/config/tooltool-manifests/macosx64/cross-releng.manifest',
|
||||||
"tooltool_url": 'http://relengapi/tooltool/',
|
"tooltool_url": 'http://relengapi/tooltool/',
|
||||||
|
"tooltool_bootstrap": "setup.sh",
|
||||||
'tooltool_script': ["/builds/tooltool.py"],
|
'tooltool_script': ["/builds/tooltool.py"],
|
||||||
'tooltool_cache': os.environ.get('TOOLTOOL_CACHE'),
|
'tooltool_cache': os.environ.get('TOOLTOOL_CACHE'),
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -10,6 +10,21 @@ from mozharness.mozilla.mock import ERROR_MSGS
|
|||||||
|
|
||||||
class Repackage(BaseScript):
|
class Repackage(BaseScript):
|
||||||
|
|
||||||
|
config_options = [[
|
||||||
|
['--signed-input', ],
|
||||||
|
{"action": "store",
|
||||||
|
"dest": "signed_input",
|
||||||
|
"type": "string",
|
||||||
|
"default": os.environ.get('SIGNED_INPUT'),
|
||||||
|
"help": "Specify the signed input (url)"}
|
||||||
|
], [
|
||||||
|
['--output-file', ],
|
||||||
|
{"action": "store",
|
||||||
|
"dest": "output_file",
|
||||||
|
"type": "string",
|
||||||
|
"help": "Specify the output filename"}
|
||||||
|
]]
|
||||||
|
|
||||||
def __init__(self, require_config_file=False):
|
def __init__(self, require_config_file=False):
|
||||||
script_kwargs = {
|
script_kwargs = {
|
||||||
'all_actions': [
|
'all_actions': [
|
||||||
@@ -20,22 +35,24 @@ class Repackage(BaseScript):
|
|||||||
}
|
}
|
||||||
BaseScript.__init__(
|
BaseScript.__init__(
|
||||||
self,
|
self,
|
||||||
|
config_options=self.config_options,
|
||||||
require_config_file=require_config_file,
|
require_config_file=require_config_file,
|
||||||
**script_kwargs
|
**script_kwargs
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Assert we have it either passed in or in environment
|
||||||
|
assert self.config.get('signed_input'), \
|
||||||
|
"Must pass --signed-input or be set in the environment as SIGNED_INPUT"
|
||||||
|
|
||||||
def download_input(self):
|
def download_input(self):
|
||||||
config = self.config
|
config = self.config
|
||||||
dirs = self.query_abs_dirs()
|
|
||||||
|
|
||||||
input_home = config['input_home'].format(**dirs)
|
url = config['signed_input']
|
||||||
|
status = self.download_file(url=url,
|
||||||
for path, url in config["download_config"].items():
|
file_name=config['input_filename'],
|
||||||
status = self.download_file(url=url,
|
parent_dir=config['input_home'])
|
||||||
file_name=path,
|
if not status:
|
||||||
parent_dir=input_home)
|
self.fatal("Unable to fetch signed input from %s" % config['signed_input'])
|
||||||
if not status:
|
|
||||||
self.fatal("Unable to fetch signed input from %s" % url)
|
|
||||||
|
|
||||||
def setup(self):
|
def setup(self):
|
||||||
self._run_tooltool()
|
self._run_tooltool()
|
||||||
@@ -52,7 +69,6 @@ class Repackage(BaseScript):
|
|||||||
dirs = {}
|
dirs = {}
|
||||||
dirs['abs_tools_dir'] = os.path.join(abs_dirs['abs_work_dir'], 'tools')
|
dirs['abs_tools_dir'] = os.path.join(abs_dirs['abs_work_dir'], 'tools')
|
||||||
dirs['abs_mozilla_dir'] = os.path.join(abs_dirs['abs_work_dir'], 'src')
|
dirs['abs_mozilla_dir'] = os.path.join(abs_dirs['abs_work_dir'], 'src')
|
||||||
dirs['output_home'] = os.path.join(abs_dirs['abs_work_dir'], 'artifacts')
|
|
||||||
for key in dirs.keys():
|
for key in dirs.keys():
|
||||||
if key not in abs_dirs:
|
if key not in abs_dirs:
|
||||||
abs_dirs[key] = dirs[key]
|
abs_dirs[key] = dirs[key]
|
||||||
@@ -62,19 +78,17 @@ class Repackage(BaseScript):
|
|||||||
def repackage(self):
|
def repackage(self):
|
||||||
config = self.config
|
config = self.config
|
||||||
dirs = self.query_abs_dirs()
|
dirs = self.query_abs_dirs()
|
||||||
|
infile = os.path.join(config['input_home'], config['input_filename'])
|
||||||
# Make sure the upload dir is around.
|
outfile = os.path.join(dirs['abs_upload_dir'], config['output_filename'])
|
||||||
self.mkdir_p(dirs['output_home'])
|
command = [sys.executable, 'mach', '--log-no-times', 'repackage',
|
||||||
|
'dmg',
|
||||||
for repack_config in config["repackage_config"]:
|
'--input', infile,
|
||||||
command = [sys.executable, 'mach', '--log-no-times', 'repackage'] + \
|
'--output', outfile]
|
||||||
[arg.format(**dirs)
|
return self.run_command(
|
||||||
for arg in list(repack_config)]
|
command=command,
|
||||||
self.run_command(
|
cwd=dirs['abs_mozilla_dir'],
|
||||||
command=command,
|
halt_on_failure=True,
|
||||||
cwd=dirs['abs_mozilla_dir'],
|
)
|
||||||
halt_on_failure=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
def _run_tooltool(self):
|
def _run_tooltool(self):
|
||||||
config = self.config
|
config = self.config
|
||||||
@@ -84,46 +98,24 @@ class Repackage(BaseScript):
|
|||||||
manifest_src = config.get('tooltool_manifest_src')
|
manifest_src = config.get('tooltool_manifest_src')
|
||||||
if not manifest_src:
|
if not manifest_src:
|
||||||
return self.warning(ERROR_MSGS['tooltool_manifest_undetermined'])
|
return self.warning(ERROR_MSGS['tooltool_manifest_undetermined'])
|
||||||
|
fetch_script_path = os.path.join(dirs['abs_tools_dir'],
|
||||||
|
'scripts/tooltool/tooltool_wrapper.sh')
|
||||||
tooltool_manifest_path = os.path.join(dirs['abs_mozilla_dir'],
|
tooltool_manifest_path = os.path.join(dirs['abs_mozilla_dir'],
|
||||||
manifest_src)
|
manifest_src)
|
||||||
cmd = [
|
cmd = [
|
||||||
sys.executable, '-u',
|
'sh',
|
||||||
os.path.join(dirs['abs_mozilla_dir'], 'mach'),
|
fetch_script_path,
|
||||||
'artifact',
|
|
||||||
'toolchain',
|
|
||||||
'-v',
|
|
||||||
'--retry', '4',
|
|
||||||
'--tooltool-manifest',
|
|
||||||
tooltool_manifest_path,
|
tooltool_manifest_path,
|
||||||
'--tooltool-url',
|
|
||||||
config['tooltool_url'],
|
config['tooltool_url'],
|
||||||
|
config['tooltool_bootstrap'],
|
||||||
]
|
]
|
||||||
auth_file = self._get_tooltool_auth_file()
|
cmd.extend(config['tooltool_script'])
|
||||||
if auth_file:
|
|
||||||
cmd.extend(['--authentication-file', auth_file])
|
|
||||||
cache = config.get('tooltool_cache')
|
cache = config.get('tooltool_cache')
|
||||||
if cache:
|
if cache:
|
||||||
cmd.extend(['--cache-dir', cache])
|
cmd.extend(['-c', cache])
|
||||||
self.info(str(cmd))
|
self.info(str(cmd))
|
||||||
self.run_command(cmd, cwd=dirs['abs_mozilla_dir'], halt_on_failure=True)
|
self.run_command(cmd, cwd=dirs['abs_mozilla_dir'], halt_on_failure=True)
|
||||||
|
|
||||||
def _get_tooltool_auth_file(self):
|
|
||||||
# set the default authentication file based on platform; this
|
|
||||||
# corresponds to where puppet puts the token
|
|
||||||
if 'tooltool_authentication_file' in self.config:
|
|
||||||
fn = self.config['tooltool_authentication_file']
|
|
||||||
elif self._is_windows():
|
|
||||||
fn = r'c:\builds\relengapi.tok'
|
|
||||||
else:
|
|
||||||
fn = '/builds/relengapi.tok'
|
|
||||||
|
|
||||||
# if the file doesn't exist, don't pass it to tooltool (it will just
|
|
||||||
# fail). In taskcluster, this will work OK as the relengapi-proxy will
|
|
||||||
# take care of auth. Everywhere else, we'll get auth failures if
|
|
||||||
# necessary.
|
|
||||||
if os.path.exists(fn):
|
|
||||||
return fn
|
|
||||||
|
|
||||||
def _get_mozconfig(self):
|
def _get_mozconfig(self):
|
||||||
"""assign mozconfig."""
|
"""assign mozconfig."""
|
||||||
c = self.config
|
c = self.config
|
||||||
|
|||||||
Reference in New Issue
Block a user