We'll be consolidating code from mach_bootstrap.py and mozboot.
We don't want mach_bootstrap.py to import bootstrap.py because it
imports nearly every module under mozboot. So establish a standalone
module with minimal dependencies to hold utility code. Move
get_state_dir() there.
MozReview-Commit-ID: Hw5VB5OZGCi
This also fixes the issue of processing the artifacts twice in some
situations (bug 1275673). Note that the artifact download no longer
happens when a specific target is passed to 'mach build'.
MozReview-Commit-ID: Ktys6u3r1kG
https://hg.mozilla.org/firefox now exists. It is a unified Firefox
repository containing the heads of all the major Firefox repos
(mozilla-central, inbound, aurora, beta, release, esrs, etc).
Having 1 unified repository is more useful and incurs less overhead
than N separate repos. We want to encourage the use of the unified
repository. So, we start cloning from it.
The unified repo on the server is configured in such a way that
manifest delta chains can become very long - over 30,000 deltas. This
can make manifest reading very slow and slow down many Mercurial
operations. The server compensates for this by setting
format.maxchainlen=10000 to limit the delta chains to 10,000.
Unfortunately, this setting is not preserved when clients do a
traditional clone: the changegroup consists of a single delta chain
and the client will use its own settings (often the default) to
break the chain. This will result in the client re-creating very long
delta chains. So, as part of initializing the new repo we configure
format.maxchainlen in its .hg/hgrc so it doesn't suffer from this
performance issue.
We could achieve the same result by passing the --config option to
`hg clone`. However, the option won't be preserved in the repo's
.hg/hgrc and subsequent `hg pull` operations could result in the
creation of very long delta chains. So we need to write the config
option in the .hg/hgrc. `hg clone` is equivalent to `hg init` +
`hg pull` anyway, so we just separate out the steps and insert a
write to .hg/hgrc in between.
We also set the "default" path (like `hg clone` would do).
DONTBUILD (NPOTB)
MozReview-Commit-ID: Fs4cz9YvdCv
I've always been bothered that the one-line bootstrap configures your system
then leaves you on the hook to clone source code and configure the build
system. I'd like the bootstrap wizard to guide you through end-to-end.
This commit addresses part of the disconnect by offering to clone the
Mercurial source repository at the end of bootstrap.
We only offer to clone if we aren't running from a Firefox source checkout
(likely the one-line bootstrap invocation) and if we are in interactive
mode.
I'd like to eventually offer Git support here. Mercurial is the canonical
repo, so it makes sense to start with that.
MozReview-Commit-ID: 6TSZwxB3702
Without this, we attempt to execute "hg" as a native Win32 program
and get the dreaded "%1 is not a valid Win32 application" error because
"hg" has a shebang and only executes inside a UNIX-like shell.
MozReview-Commit-ID: 5sUrxh1IxFC
The buildconfig module predates MozbuildObject.from_environment, and
it's about time to start factoring things out such that we only have
one way to get config.status data. This is step 1: making the
buildconfig module use MozbuildObject.from_environment.
Eventually, we'll want to remove the buildconfig module uses everywhere.
The topobjdir-finding logic in mozbuild relies on MOZ_CURRENT_PROJECT
being set, and it's currently only set when going through client.mk.
On automation, during universal builds, make check is invoked directly
in one of the objdirs, so MOZ_CURRENT_PROJECT is not set. We've had
other similar problems in the past. Ensuring MOZ_CURRENT_PROJECT is set
in the objdir itself should reduce the risk of other such issues in the
future.
Historically, a mozinfo for js standalone build has not been necessary,
but with the move towards a) having things work with mach and b)
buildconfig using the MozbuildObject.from_environment in next patch,
mozinfo becomes necessary (it's required for
MozbuildObject.from_environment to find the directory is an objdir).
Interestingly, hazard builds do both a js standalone build and a desktop
Firefox build at the same time, both of which are done with MOZCONFIG
set in the environment... with the Firefox mozconfig. The result of now
writing a mozinfo for js standalone builds is that in that case, they
end up with a reference to the mozconfig, which the build system then
reads, and finds a MOZ_OBJDIR, which then doesn't match the js
standalone build objdir. So for those builds, reset MOZCONFIG.
This makes it easier to lint a path that otherwise wouldn't have been linted due to the include/exclude
directives. Now, you can pass in -n/--no-filter instead of needing to modify the linter configuration file.
MozReview-Commit-ID: GMJuE2C1NyY
The wizard has been ported to the version-control-tools repository
and in-tree consumers have been switched to consume it from there. This
code is now dead. Kill it.
References to the now-defunct code have been removed/updated.
MozReview-Commit-ID: 5fpCXdNIp8L
The Mercurial setup wizard has now been ported to the version-control-tools
repository, where it has testing and integrates better with Mercurial
configs.
The bootstrapper has been taught how to invoke the new version of the
Mercurial setup wizard.
This commit switched `mach mercurial-setup` to call the bootstrapper
code for invoking the Mercurial setup wizard from version-control-tools.
As of this commit, the Mercurial setup wizard in tools/mercurial is
unused.
MozReview-Commit-ID: 3xzgOYZWTZn
If a state directory is available and we're running in interactive mode
(or have been told otherwise), we now configure Mercurial during
bootstrap.
This consists of cloning version-control-tools to the state directory
(mimicking code in `mach mercurial-setup` today) and running the
config wizard from version-control-tools.
Code for cloning/updating repositories has been stolen from
tools/mercurial/hgsetup.
As the inline TODO notes, I'd like to eventually support
configuring Git during bootstrap. Since Mercurial is the canonical VCS
for Firefox and since we already have a Mercurial setup wizard (and
don't have a Git one yet), I don't think we should block on implementing
Git support.
MozReview-Commit-ID: 1FZyWIlHZNy
Currently, on first run of `mach` it prompts you to create a state
directory. The hand-off between bootstrap and `mach` has always
bothered me because bootstrap is supposed to get your system in a good
state.
In this commit, we teach the bootstrap tool to create the state
directory when not present. This duplicates functionality from `mach`.
The justification for the duplication is explained by inline comment.
In future commits, we'll build on this work to have the bootstrapper
run the Mercurial config wizard, which needs this state directory.
MozReview-Commit-ID: CPKVuRJ3peM
This begins the consolidation of `mach mercurial-setup` into
`mach bootstrap`. The first step is to move the content of the
mach_commands.py file into the bootstrapper's.
I'm not crazy about adding the sys.path entry for tools/mercurial.
I intend to clean this up later.
MozReview-Commit-ID: Cq56wPG8sO1
This commit simply moves 'testing/eslint' to 'tools/lint/eslint' and the eslint related
mach command from 'python/mach_commands.py' to 'tools/lint/mach_commands.py'. It shouldn't
have any functional change on running eslint, either through mach or taskcluster.
This is in preparation for bug 1258341, to make the diffs there a little easier to read.
MozReview-Commit-ID: K03sn9lv9Lv
Killing the sccache background daemon is part of postflight_all, but in
the current setup, postflight_all happens at the end of a "normal" build,
but we run automation build steps after that.
What happens then is that more compilations happen (gtests), which start
sccache again, but there's nothing to kill sccache again once this is
all done.
Now that the OSX universal builds postflight is gone, it is not
necessary for postflight_all to happen before the automation build steps.
So ensure postflight_all scripts happen last.
The downside of this change is that this now prevents sccache.log from
being uploaded, but we should probably send processed data to the graph
server instead.
Killing the sccache background daemon is part of postflight_all, but in
the current setup, postflight_all happens at the end of a "normal" build,
but we run automation build steps after that.
What happens then is that more compilations happen (gtests), which start
sccache again, but there's nothing to kill sccache again once this is
all done.
Now that the OSX universal builds postflight is gone, it is not
necessary for postflight_all to happen before the automation build steps.
So ensure postflight_all scripts happen last.
The downside of this change is that this now prevents sccache.log from
being uploaded, but we should probably send processed data to the graph
server instead.
The correct version of Python will get installed from the install_python method instead of with the system packages.
This is more in-line with how a bootstrapper *should* extend from the base bootstrapper.
MozReview-Commit-ID: JIMGF7XKL02
Overrode BaseBootstrapper.which to append '.exe' to any which checks since (hopefully) anything the bootstrapper looks for, must be a windows executable.
Changed base bootstrapper class to use str instead of unicode to avoid a bug in the MinGW version of Python where subprocces.Popen will not accept environment variables that are in unicode instead of str.
MozReview-Commit-ID: 4m8xNifawYS
Added convenience method for installing from pip.
Windows bootstrapper implements upgrade_mercurial to install mercurial from pip.
MozReview-Commit-ID: ClqNA2NmQcc
Windows bootstrapper checks if pacman is installed before continuing.
Added a convenience method similar to BaseBootstrapper.which that works with the mingw version of python in msys2.
MozReview-Commit-ID: 6AG2c18KF0U
These new convenience methods let the bootstrapper update the local package list, upgrade all installed packages, and install new packages.
MozReview-Commit-ID: KZPyBl0OU6Z
Created a WindowsBootstrapper class that raises a NotImplementedError when initialized.
As WindowsBootstrapper is implemented, set $MOZ_WINDOWS_BOOTSTRAP to '1' in your environment to test it.
Bootstrapper now detects if the system is being run on Windows, and if it is dispatches to the WindowsBootstrapper.
MozReview-Commit-ID: 3x6PDPuLtzs
This is a really simple and ugly formatter that is compatible with
treeherder's error highlighting mechanism. It is designed to be identical
to the current eslint output on treeherder:
https://dxr.mozilla.org/mozilla-central/rev/4d63dde701b47b8661ab7990f197b6b60e543839/tools/lint/eslint-formatter.js
Eventually eslint will also use this and we can remove that file. Once
bug 1276486 is fixed, we can make this look a little nicer. But for now
it gets the job done.
MozReview-Commit-ID: CwfWPcwWFxF
This is a really simple and ugly formatter that is compatible with
treeherder's error highlighting mechanism. It is designed to be identical
to the current eslint output on treeherder:
https://dxr.mozilla.org/mozilla-central/rev/4d63dde701b47b8661ab7990f197b6b60e543839/tools/lint/eslint-formatter.js
Eventually eslint will also use this and we can remove that file. Once
bug 1276486 is fixed, we can make this look a little nicer. But for now
it gets the job done.
MozReview-Commit-ID: CwfWPcwWFxF
These variables specify a version of Mercurial that is considered
modern and won't trigger giant warnings about being out of date.
We bump to 3.7.3 because 3.7.3 contains security fixes and it is
important for as many users as possible to get these security fixes.
We also update the messaging to indicate security issues with older
releases.
MozReview-Commit-ID: H4utKINrW0V
Now that the VisualStudio backend will no-op if nothing has changed, it
should be safe to always run this backend.
On first run, backend generation takes ~3.5s on my machine. On subsequent run,
it takes ~1.5s. Wall time for a no-op config.status is now ~15.7s. We could like
make the Visual Studio backend faster by not writing so many project files.
But this would require consolidating libraries in moz.build files. And that's
out of scope for this change.
We drop the check for MSVS_VERSION because it won't always be defined on
MinGW/GCC builds. We simply default to "2015" if it isn't set.
MozReview-Commit-ID: 5W38HMGmcuV
