The only tricky piece here is that the resulting toolchain archive is
private, and uses a newly allocated Task Cluster scope
(queue:get-artifact:project/gecko/android-sdk/*) to restrict access to
the archive. All SCM levels (1, 2, 3) have been given the new scope:
see https://tools.taskcluster.net/auth/roles/moz-tree:level:1 and
friends.
MozReview-Commit-ID: CcDqDOHODpe
The only tricky piece here is that the resulting toolchain archive is
private, and uses a newly allocated Task Cluster scope
(queue:get-artifact:project/gecko/android-sdk/*) to restrict access to
the archive. All SCM levels (1, 2, 3) have been given the new scope:
see https://tools.taskcluster.net/auth/roles/moz-tree:level:1 and
friends.
MozReview-Commit-ID: CcDqDOHODpe
When adding sccache toolchain jobs in bug 1381772, building with gcc
failed, and building with clang worked, so I just went with the path of
least resistance. That's however a suboptimal position in the dependency
graph, so it's still preferable to use gcc if possible.
Looking exactly how it fails, it turns out it's because without CC being
set, ring wants to build with "cc", which ends up being the system gcc
instead of ours (our gcc archive doesn't provide "cc", only "gcc"), and
it is too old to support the compiler flags ring uses.
So setting CC does the trick.
Copy the repack_rust.py from the rust-build docker container
so it can be used more generally by other taskcluster jobs.
Add --host, --target, and --suffix switches, allowing control
of the packaged toolchain and standard library builds from
the command line.
This drops the previous default behaviour of packaging all
supported platforms and targets.
Add a hard-coded copy of the Rust release signing key to
the script and add it to the running user's gpg config
so we can validate downloaded artifacts from the project
in automation.
Remove the keybase artifact validation since it requires
out-of-project network services and doesn't provide much
advantage in automation.
Calculate the SHA-2 checksum during download and remove
the dependency on shasum/sha256sum command-line tools.
Use more python for filesystem an process interaction
in general.
Create a generic rustc.tar.* package to correctly match
the unversioned unpack dirctory name.
Add support for copying the package to an output directory
if the UPLOAD_DIR environment variable is set. This lets
us hook up the script to taskcluster toolchain jobs without
an external wrapper.
MozReview-Commit-ID: 68LmY3QVU8V
Our current sccache build links in openssl's libraries dynamically. The
sonames of the dynamic libraries linked in are specific to the
CentOS/Fedora-ish systems that we build on; attempting to run the
generated sccache binaries on different systems (e.g. Debian-ish) will
result in failure. All of our current automation images are
CentOS-based, but for various reasons, Debian-based images may be used
in the future, and it would be great to have an sccache binary to run on
such systems as well. (It might also be interesting to distribute the
sccache binary we use to local developers as well, but that's a bit
further off.)
Therefore, this patch alters the sccache build on Linux to use static
linking for openssl. We cannot use the system openssl we build on
because the system openssl links to libkrb5, and the distribution we use
for the system images does not provide static libraries of libkrb5.
Building openssl ourself enables us to eliminate the libkrb5 dependency.
An sccache binary from builds with this patch depends on the following
libraries:
froydnj@hawkeye:~$ ldd sccache2/sccache
linux-vdso.so.1 => (0x00007ffe02b39000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff0e7403000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff0e71fb000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff0e6fdd000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff0e6dc6000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff0e69fc000)
/lib64/ld-linux-x86-64.so.2 (0x0000557c8540b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff0e66f2000)
which are standard on any Linux distribution.
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: 554IPMRWgzK
While this looks like going backwards, it is desirable to build clang
against GCC 4.8, such that it contains its libgcc. This, in turn, will
solve problems using clang 3.9 with static-analysis builds (details in
bug 1356926). Another way to fix those problems would be to build clang
3.8 but that too would require GCC 4.8. Upgrading those builds to clang
3.9 will also allow to enable stylo on them.
It becomes a library of some sort, so that multiple scripts can benefit
from it to build different versions of GCC.
The GPG key associated with GCC is also refreshed from keys.gnupg.net,
adding a new subkey, used to sign newer versions of GCC (and
postprocessed with pgpstrip to make it smaller).
We're soon going to build multiple versions of clang and gcc for linux,
and we need to differentiate them. Furthermore, there is a need for the
base-toolchains builds to use a fixed version of clang and gcc. So
rename the clang and gcc toolchain jobs to include their version, add
aliases to satisfy all existing jobs, and adjust the base-toolchains
jobs to use the explicit version.
With the support added in bug 1382564, toolchains can be downloaded
without a tooltool manifest at all, and it's desirable to get rid of
tooltool manifests on jobs that don't need one.
Bug 1374940 adds a MOZ_TOOLCHAINS environment variable with a list of
path@task-id strings, where task-id is corresponding to the (possibly
optimized) toolchain job, and path corresponding to the
toolchain-artifact defined for that toolchain job.
We want to use that to pull artifacts instead of tooltool packages.
The scripts that use tooltool-download.sh don't run regularly, but when
they do, they might hit some download problems (the relengapi proxy
tends to be rather unreliable for some reason), and in that case, it
would be better to retry a few times, like other job types, rather than
fail directly.
All the current users of tooltool-download.sh set $WORKSPACE. This will
allow to reuse the script on different types of workers, that don't have
$WORKSPACE set to $HOME/workspace, but still have the source in
$WORKSPACE/build/src.
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: zehcGJrUQX
The toolchain build scripts are currently defining the tooltool
manifest they use on their own. We move the definitions to the
taskcluster job definitions to normalize on everything using that.
While here, since the tooltool call from the cctools build scripts was
verbose, make the one in tooltool-download.sh verbose. It will benefit
the other things using tooltool-download.sh, making debugging easier
when one needs to look at the tooltool download logs...
