This merely extends the logic involved in shipping language packs to AMO. All devedition language packs will be shipped as 'unlisted' for now, meaning that there is no extra AMO work involved. The extension ID is taken from the langpack itself.
Differential Revision: https://phabricator.services.mozilla.com/D1104
On ESR[60] stub installer isn't designed to work, and we expect most users of esr will have no use for a stub installer. However this poses some problems where the taskgraph assumes that any Nightly generates a stub installer and thus it should be available along the way.
The patch hardcodes the list of branches that do not need a stub installer, though we'll want to clean up this specification in some way, as Thunderbird will likely need it to be cleaner and we may want on-change builds to use this logic (e.g. for on-change l10n).
Differential Revision: https://phabricator.services.mozilla.com/D1082
There are now only a handful of buildbot jobs remaining and the concern over
outdated treeherder exclusion profiles has largely been resolved.
This does remove the tc() group from a substantial number of tasks which will
now show up as top level tasks, potentially adding clutter. In some cases, we
might want to re-add a new group (e.g group builds or compiled tests together).
However rather than try to predict the best group names for tasks I'm unfamiliar
with, I think it's best to land this as is. Then if things are looking too
cluttered at the root namespace, file follow-up bugs as needed.
MozReview-Commit-ID: 8SMwjDwAOzV
Other related changes:
* Only target.{zip,tar.gz,tar.bz} get the widevine signature (not the various installers).
* Linux{32,64}-nightly are now repackaged. Their mar files are not signed during signed step anymore. It now happens after the repackage.
* As a consequence, funsize routes for linux are now set to repackage-signing (instead of signing)
* Signed upstream artifacts are now defined in a dedicated module (to avoid duplication)
* Platforms defined in beetmover_repackage now allow regex (to reduce duplication too)
* Mozharness configs: Delete unused (and misleading) `src_mozconfig` for windows. This value is actually not used when `run_configure` (in the same dict) is set to False.
MozReview-Commit-ID: COKqevW9Mzn
Land date changes to support windows nightlies onto central
Supports beetmoving from the build-signing task, as well as ignoring dependencies if we have nothing to beetmove in them (e.g. build-signing for OSX)
MozReview-Commit-ID: 24byn1posKT
Several years ago there was a single zip file for all test files. Clients
would only extract the files they needed. Thus, zip was a reasonable
archive format because it allowed direct access to members without
having to decompress the entirety of the stream.
We have since split up that monolithic archive into separate,
domain-specific archives. e.g. 1 archive for mochitests and one
for xpcshell tests. This drastically cut down on network I/O
required on testers because they only fetched archives/data that
was relevant. It also enabled parallel generation of test archives,
we shaved dozens of seconds off builds due to compression being
a long pole.
Despite the architectural changes to test archive management, we
still used zip files. This is not ideal because we no longer access
specific files in test archives and thus don't care about single/partial
member access performance.
This commit implements support for generating tar.gz test archives.
And it switches the web-platform archive to a tar.gz file.
The performance implications for archive generation are significant:
before: 48,321,250 bytes; 6.05s
after: 31,844,267 bytes; 4.57s
The size is reduced because we have a single compression context
so data from 1 file can benefit compression in a subsequent file.
CPU usage is reduced because the compressor has to work less with
1 context than it does with N. While I didn't measure it, decompression
performance should also be improved for the same reasons. And of course
network I/O will be reduced.
mozharness consumers use a generic method for handling unarchiving.
This method automagically handles multiple file extensions. So as long
as downstream consumers aren't hard coding ".zip" this change should
"just work."
MozReview-Commit-ID: LQa5MIHLsms
Several years ago there was a single zip file for all test files. Clients
would only extract the files they needed. Thus, zip was a reasonable
archive format because it allowed direct access to members without
having to decompress the entirety of the stream.
We have since split up that monolithic archive into separate,
domain-specific archives. e.g. 1 archive for mochitests and one
for xpcshell tests. This drastically cut down on network I/O
required on testers because they only fetched archives/data that
was relevant. It also enabled parallel generation of test archives,
we shaved dozens of seconds off builds due to compression being
a long pole.
Despite the architectural changes to test archive management, we
still used zip files. This is not ideal because we no longer access
specific files in test archives and thus don't care about single/partial
member access performance.
This commit implements support for generating tar.gz test archives.
And it switches the web-platform archive to a tar.gz file.
The performance implications for archive generation are significant:
before: 48,321,250 bytes; 6.05s
after: 31,844,267 bytes; 4.57s
The size is reduced because we have a single compression context
so data from 1 file can benefit compression in a subsequent file.
CPU usage is reduced because the compressor has to work less with
1 context than it does with N. While I didn't measure it, decompression
performance should also be improved for the same reasons. And of course
network I/O will be reduced.
mozharness consumers use a generic method for handling unarchiving.
This method automagically handles multiple file extensions. So as long
as downstream consumers aren't hard coding ".zip" this change should
"just work."
MozReview-Commit-ID: LQa5MIHLsms
Several years ago there was a single zip file for all test files. Clients
would only extract the files they needed. Thus, zip was a reasonable
archive format because it allowed direct access to members without
having to decompress the entirety of the stream.
We have since split up that monolithic archive into separate,
domain-specific archives. e.g. 1 archive for mochitests and one
for xpcshell tests. This drastically cut down on network I/O
required on testers because they only fetched archives/data that
was relevant. It also enabled parallel generation of test archives,
we shaved dozens of seconds off builds due to compression being
a long pole.
Despite the architectural changes to test archive management, we
still used zip files. This is not ideal because we no longer access
specific files in test archives and thus don't care about single/partial
member access performance.
This commit implements support for generating tar.gz test archives.
And it switches the web-platform archive to a tar.gz file.
The performance implications for archive generation are significant:
before: 48,321,250 bytes; 6.05s
after: 31,844,267 bytes; 4.57s
The size is reduced because we have a single compression context
so data from 1 file can benefit compression in a subsequent file.
CPU usage is reduced because the compressor has to work less with
1 context than it does with N. While I didn't measure it, decompression
performance should also be improved for the same reasons. And of course
network I/O will be reduced.
mozharness consumers use a generic method for handling unarchiving.
This method automagically handles multiple file extensions. So as long
as downstream consumers aren't hard coding ".zip" this change should
"just work."
MozReview-Commit-ID: LQa5MIHLsms
This is a more robust approach than using substring matching on task labels.
As an optimization, this simply avoids creating balrog tasks for unsigned beets
using only-for-attributes, rather than omitting them in a transform.
MozReview-Commit-ID: 8MNOxu0WgXo
