We've had this restriction for a while (with a note of it in the docs,
but the note in the docs can easily be overlooked, leading to confusing
errors). Erroring out right on Mach startup and providing instructions
for the workaround immediately should help users get around this issue
faster.
Differential Revision: https://phabricator.services.mozilla.com/D183209
This makes loading almost all commands faster, since only one module
file is loaded rather than all of them. There is one main exception,
dealing with 'help'. Running `./mach help` (or -h or --help) requires
the description text for every command, so every module file is still
loaded.
We could expand this improvement here to consolidate all commands and
their parameters in this `MACH_COMMANDS` dict, but the only two benefits
are improving help, and not having two places where the commands are
specified (their file, and this dict).
There's a lot of extra work needed to do that, especially for handling
sub commands, and it did not seem worth the cost for the benefit at this
time.
Depends on D180499
Differential Revision: https://phabricator.services.mozilla.com/D180500
This activated virtualenv for a command is managed
`CommandSiteManager` and it is passed down to where it was activated
before to prevent a second, redundant, activation.
Differential Revision: https://phabricator.services.mozilla.com/D180499
Also add the associated virtualenv names to each command that has one in
preparation for loading the virtualenvs earlier (next patch).
Differential Revision: https://phabricator.services.mozilla.com/D181032
This reverts bug 1801826 and instead prepends the msys2 path only for
the build, which is a bit more sensible.
This allows us to remove the existing warning from old_configure, since
that can't happen now.
Differential Revision: https://phabricator.services.mozilla.com/D163073
This reverts bug 1801826 and instead prepends the msys2 path only for
the build, which is a bit more sensible.
This allows us to remove the existing warning from old_configure, since
that can't happen now.
Differential Revision: https://phabricator.services.mozilla.com/D163073
This makes sure that we find the right tools in the right msys2
environment in case we get called from another msys2 env.
Differential Revision: https://phabricator.services.mozilla.com/D162753
Added `mach uniffi generate` which executes `uniffi-bindgen-gecko-js` to
generate UniFFI bindings. It's unfortunate that we need to check these
files in, but I couldn't figure out a way to auto-generate them as part
of the build process.
Adding `#include "nsIContent.h"` to dom/base/nsINodeList.h. I think
that should have been present before, but things built okay because of
the way things got combined in the uniffied .cpp files. Adding these new
webIDL files bumped `NodeListBinding.cpp` to a new uniffied .cpp file
which caused the build to fail.
Differential Revision: https://phabricator.services.mozilla.com/D144468
Added `mach uniffi generate` which executes `uniffi-bindgen-gecko-js` to
generate UniFFI bindings. It's unfortunate that we need to check these
files in, but I couldn't figure out a way to auto-generate them as part
of the build process.
Adding `#include "nsIContent.h"` to dom/base/nsINodeList.h. I think
that should have been present before, but things built okay because of
the way things got combined in the uniffied .cpp files. Adding these new
webIDL files bumped `NodeListBinding.cpp` to a new uniffied .cpp file
which caused the build to fail.
Differential Revision: https://phabricator.services.mozilla.com/D144468
As per the docs:
> Return the number of logical CPUs in the system (same as os.cpu_count in Python 3.4) or None if undetermined.
> [...] On OpenBSD and NetBSD psutil.cpu_count(logical=False) always return None
Glean tries to coerce values to the expected type. `None` cannot be
coerced to an integer, so this blows up.
We conditionally report that value now only if we can properly detect
one.
Differential Revision: https://phabricator.services.mozilla.com/D149201
When using Mach outside of MozillaBuild, rather than avoiding adding
paths entirely if any part of MozillaBuild is already in the `PATH`,
instead conditionally add each one if it doesn't exist already.
This ensures no duplication of paths, while also correctly handling the
not-uncommon case of developers manually adding MozillaBuild's Python to
their Windows `PATH`.
Differential Revision: https://phabricator.services.mozilla.com/D141377
Windows uses `;` as the path separator, Unix-based systems use `:`.
`os.pathsep` conveniently represents the current system's separator.
All OS-specific path separators have been replaced with `os.pathsep`
except for some explicitly Windows-specific mozharness configs.
Differential Revision: https://phabricator.services.mozilla.com/D140981
This reduces some code duplication and paves the way for modifying how
Mach loads modules.
The plan is to follow this up by making this function take a more fleshed out 'spec'
that maps command names to the files they're defined at and possibly more metadata.
Since this may affect how Mach works internally (e.g. with lazy loading in the roadmap),
it makes sense to move this logic inside the Mach class.
Differential Revision: https://phabricator.services.mozilla.com/D136789
Assuming that the `MOZILLABUILD` environment variable is set, allow
invoking Mach from non-MozillaBuild terminals.
Note that MozillaBuild still needs to be installed, and the
`MOZILLABUILD` environment variable will have to be set.
For future reference: when I tried setting this up with Windows
Store's Python 3.9, I encountered issues when running binaries installed
via `pip`: it would fail with `abort: failed to load Python DLL
python3x.dll`.
Differential Revision: https://phabricator.services.mozilla.com/D133936
Move Python version check as early as possible so that more code can
safely depend on modern behaviour while out-of-date Python versions still
get graceful error messages.
Without this change, Python 2 usages fail on importing `importlib.util`
before the nice "out-of-date version" warning is printed.
Differential Revision: https://phabricator.services.mozilla.com/D134185
Move Python version check as early as possible so that more code can
safely depend on modern behaviour while out-of-date Python versions still
get graceful error messages.
Without this change, Python 2 usages fail on importing `importlib.util`
before the nice "out-of-date version" warning is printed.
Differential Revision: https://phabricator.services.mozilla.com/D134185
We only need a workdir-scoped state_dir when an on-disk virtualenv will
be created for the Mach site.
This change defers the resolution of the state_dir until we know that a
VENV will be created.
Also modify "telemetry.py" so that it isn't creating a
scoped state-dir to compare "sys.executable" against.
Differential Revision: https://phabricator.services.mozilla.com/D132706
Build and run the Mach virtualenv from a `state_dir` that is
"specific-to-topsrcdir".
As part of this, move `get_state_dir()` to `mach` so that it's usable
before `sys.path` entries are fully set up.
Differential Revision: https://phabricator.services.mozilla.com/D130383
Consolidate Mach virtualenv management to the front of the
Mach process. This obsoletes `./mach create-mach-environment`
and simplifies the `sh` portion of the top-level `./mach` script.
This helps ensure that the Mach virtualenv doesn't become
out-of-sync and simplifies the mental model of the Mach
virtualenv situation.
Differential Revision: https://phabricator.services.mozilla.com/D120401
At Mach initialization time, it needs to know if it's not running from
the Mach virtualenv so it can decide to insulate itself from the
external sys.path.
However, the current mechanism of detecting the Mach virtualenv
struggles if the venv is old, and therefore missing its metadata file.
In such a case, it's recognized as "not the Mach venv" rather than "an
out-of-date Mach venv".
As part of this, I'm realizing that simply using the metadata file as a
"virtualenv has finished building" marker is insufficient, because it
will cause similar confusion here. This is solved with a "finalization"
key in in the metadata.
Differential Revision: https://phabricator.services.mozilla.com/D130794
The command site manager needs to be able to do ad-hoc pip
installations, while the Mach site manager needs to manage
the system `sys.path` and conditionally create an on-disk
virtualenv.
By splitting the class into two, we can now give each use case the
attention it deserves.
Differential Revision: https://phabricator.services.mozilla.com/D129529
Sorry for the flip-flop on technique here :S
`validate_environment_packages()` was originally run when checking if a
site is up-to-date to ensure that ad-hoc pip installs didn't replace
needed packages with those of different versions.
However, since it was added, a few notes have come up:
1. The case where requirements change isn't caught by this - that is
caught earlier by the cheap "a requirements file has changed on-disk"
check.
2. This is really slow, and doing it for most Mach commands is not worth
it (as evident by how the `skip_pip_package_check` was already added
for the Mach site's use case).
3. Since the tree as-is doesn't have (common) cases where ad-hoc
installations break an environment, then this check, though helpful,
isn't adding a significant amount of value considering its performance
cost.
However, these aren't to say that this won't be valuable in the future:
I'd like to reach a point where sites are considered "sealed" by
default: no ad-hoc pip installations are allowed.
However, add the ability to mark sites as unsealed/"allowing
ad-hoc pip installations". Then, re-add the pip package check, but only
for such flexible, unsealed virtualenvs.
Differential Revision: https://phabricator.services.mozilla.com/D129692
The existing terminology had two issues:
* `VirtualenvManager` wasn't always associated with an on-disk
`virtualenv`: for example, when running in automation, Mach
"activates" a `VirtualenvManager`, updating its import scope,
but without ever creating an on-disk `virtualenv`.
* An upcoming patch splits the `VirtualenvManager` class, pulling
"on-disk virtualenv-handling functions" from the project-wide
interface for managing Python's import scope.
After some good discussion with Ahal, I think we've struck
the terminology that handles this distinction well: we'll call
the "import scope"-handling part the "site", and we'll continue
to call on-disk virtualenvs (and their representative classes)
as, well, virtualenvs.
Differential Revision: https://phabricator.services.mozilla.com/D130391
There were a bunch of locations where we were doing path shenanigans
with `requirements.pth/.vendored` items.
There was a bit of complexity because we were specifically making each
`pthfile` line be a relative path to support moving the Firefox
topsrcdir without causing issues.
However, now that we're more intelligent about checking if `pthfile`
lines are up-to-date (and since moving your topsrcdir will still require
re-generating the Mach virtualenv), this behaviour became less useful.
So, generalize `MachEnvRequirements` -> "sys.path lines" logic and
reuse it everywhere.
Differential Revision: https://phabricator.services.mozilla.com/D129693
The previous behaviour was to:
* Never add a `pthfile` to the Mach virtualenv, and
* Always add Mach's paths to the `sys.path` when Mach initializes
However, this meant that `pip` would needlessly install packages
that already exist in the vendored environment.
Tweak `pth` behaviour so that `pip` behaves more efficiently.
Differential Revision: https://phabricator.services.mozilla.com/D120402
As we leverage the Mach environment more, it becomes increasingly
important that it isn't out-of-date on developer machines.
Add an `up_to_date()` check during Mach initialization.
To minimize the cost to startup, I'm skipping the "pip list" check.
This change required moving `virtualenv` from `mozbuild` to `mach` to
make it available during the early stage of Mach init.
Differential Revision: https://phabricator.services.mozilla.com/D127144
