corysanin/tubestation
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Bug 1341214 - Add a small API to handle taskcluster queue and index requests. r=dustin

Browse Source 
Various modules under taskcluster are doing ad-hoc url formatting or
requests to taskcluster services. While we could use the taskcluster
client python module, it's kind of overkill for the simple requests done
here. So instead of vendoring that module, create a smaller one with
a limited set of functions we need.

This changes the behavior of the get_artifact function to return a
file-like object when the file is neither a json nor a yaml, but that
branch was never used (and was actually returning an unassigned
variable, so it was broken anyways).

At the same time, make the function that does HTTP requests more
error-resistant, using urllib3's Retry with a backoff factor.

Also add a function that retrieves the list of artifacts, that while
currently unused, will be used by `mach artifact` shortly.
This commit is contained in:
Mike Hommey
2017-02-17 12:04:48 +09:00
parent eb5916da95
commit c7945d8ea1
11 changed files with 110 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
taskcluster/taskgraph/docker.py
View File

@@ -12,31 +12,32 @@ import sys
import subprocess
import tarfile
import tempfile
import urllib2
import which
from subprocess import Popen, PIPE
from io import BytesIO
from taskgraph.util import docker
from taskgraph.util.taskcluster import (
find_task_id,
get_artifact_url,
)
from . import GECKO
INDEX_URL = 'https://index.taskcluster.net/v1/task/' + docker.INDEX_PREFIX + '.{}.{}.hash.{}'
ARTIFACT_URL = 'https://queue.taskcluster.net/v1/task/{}/artifacts/{}'
DOCKER_INDEX = docker.INDEX_PREFIX + '.{}.{}.hash.{}'
def load_image_by_name(image_name, tag=None):
context_path = os.path.join(GECKO, 'taskcluster', 'docker', image_name)
context_hash = docker.generate_context_hash(GECKO, context_path, image_name)
image_index_url = INDEX_URL.format('level-3', image_name, context_hash)
print("Fetching", image_index_url)
task = json.load(urllib2.urlopen(image_index_url))
index_path = DOCKER_INDEX.format('level-3', image_name, context_hash)
task_id = find_task_id(index_path)
return load_image_by_task_id(task['taskId'], tag)
return load_image_by_task_id(task_id, tag)
def load_image_by_task_id(task_id, tag=None):
artifact_url = ARTIFACT_URL.format(task_id, 'public/image.tar.zst')
artifact_url = get_artifact_url(task_id, 'public/image.tar.zst')
result = load_image(artifact_url, tag)
print("Found docker image: {}:{}".format(result['image'], result['tag']))
if tag: