This is a work-around until Bug 1405889 is deployed. Using the /bewit
endpoint does have the advantage of avoiding another issue in
taskcluster-proxy, namely that the /bewit approach streams. Fetching
through the proxy does not stream from the upstream resource; the
upstream resource is fetched and stored in taskcluster-proxy's memory,
increasing operational costs.
MozReview-Commit-ID: 8yS7zKLALhd
When a response has unconsumed content, requests closes the connection
instead of releasing it to the connection pool, such that a new
connection has to be created when another request to the same host is
done within the same Session.
When calling response.raise_for_status, the response content is left
unconsumed, meaning that without manually consuming the content first,
every time raise_for_status raises, a connection is closed.
While here, we move the raise_for_status to the _do_request function, so
that there's only one place where this is dealt with.
Various modules under taskcluster are doing ad-hoc url formatting or
requests to taskcluster services. While we could use the taskcluster
client python module, it's kind of overkill for the simple requests done
here. So instead of vendoring that module, create a smaller one with
a limited set of functions we need.
This changes the behavior of the get_artifact function to return a
file-like object when the file is neither a json nor a yaml, but that
branch was never used (and was actually returning an unassigned
variable, so it was broken anyways).
At the same time, make the function that does HTTP requests more
error-resistant, using urllib3's Retry with a backoff factor.
Also add a function that retrieves the list of artifacts, that while
currently unused, will be used by `mach artifact` shortly.
Various modules under taskcluster are doing ad-hoc url formatting or
requests to taskcluster services. While we could use the taskcluster
client python module, it's kind of overkill for the simple requests done
here. So instead of vendoring that module, create a smaller one with
a limited set of functions we need.
This changes the behavior of the get_artifact function to return a
file-like object when the file is neither a json nor a yaml, but that
branch was never used (and was actually returning an unassigned
variable, so it was broken anyways).
At the same time, make the function that does HTTP requests more
error-resistant, using urllib3's Retry with a backoff factor.
Also add a function that retrieves the list of artifacts, that while
currently unused, will be used by `mach artifact` shortly.
