Compiling C/C++/asm just requires the list of sources from _process_unified_sources() and the
Sources/GeneratedSources objects, along with the ComputedFlags and
PerSourceFlags.
The assembler invocation will need to be tweaked to support yasm as
well, but this works so far for the xpcom directory.
MozReview-Commit-ID: 91BSKV9XPLU
Detecting the upstream repository isn't always possible. Using 'git merge-base'
seems to be a more reliable way of detecting outgoing files.
MozReview-Commit-ID: 3zOdTRIcggw
In addition to help fix this bug, having the base revision will allow us
to download the most likely relevant artifacts from CI. For example, in
|mach try| we can search for a 'full-taskgraph.json' artifact from the
most relevant decision task, even if not updated to the latest central.
The ability to find the base revision will also be needed when merging
tools/tryselect/vcs.py into mozversioncontrol.
MozReview-Commit-ID: 2WRRs4AoXcK
Our toolchain detection logic checks whether we can reuse the target
C (resp. C++) compiler for the host compiler. This is generally only
applicable in the not-cross-compiling case, but we had special logic to
check for clang in the cross-compiling case and accept it, as clang is
able to generate code for multiple architectures from a single compiler
binary.
Our recent switch to clang on Android has exposed a problem in this
logic: we would never check whether the target clang, compiling for the
host, could actually find the host's headers. This was especially
problematic on OS X hosts, where the host clang contains special logic
to grovel inside the XCode installation to find C++ headers. The clang
from the NDK, however, was ignorant of the XCode installation.
Therefore, the NDK clang would happily compile code for the host, even
including C headers for the host, but would be hopelessly lost when it
came to compiling C++ headers during the actual build.
In hopes of mitigating this, we now include a check for a representative
header for C and C++ when checking compilers for each of those
languages. This check will detect such problems as the above, and will
also alert people to potentially misconfigured compilers in other
situations.
We need to modify our test framework to cope with headers being
included, since our mock environment isn't actually equipped with a full
set of compilers and headers.
b6adf66f34c6 (bug 1416052) changed the value for "fh" when this code
is called. It can now be an io.BytesIO. This type enforces that
arguments are bytes and doesn't perform automatic type coercion like
most other parts of Python 2.
self.topobjdir is a unicode. And unicode_literals isn't in effect
in this file. So convert self.topobjdir to bytes to make BytesIO
happy.
MozReview-Commit-ID: LrWTKFp3ZKT
The clobber logic is already written in Python. Now that we
always use mach in front of client.mk, we can check the clobber
state before we execute client.mk.
Since we always check the clobber state, we can remove the
CLOBBER files from various dependencies in client.mk. The
clobberer code should ensure everything is in a good state.
The refactor of the clobber Python code required some changes to
its testing. We drop some support for verifying output strings.
But testing this correctly would require a bit of effort. I don't
think it is worth it.
MozReview-Commit-ID: 69CoImCgtNm
Now that mach is being used to invoke client.mk, we can perform
objdir directory creation there.
Removing the use of mkdir_deps meant that we could remove some
included make files which AFAICT were only used to provide
$(mkdir_deps).
MozReview-Commit-ID: 4ZRToz8xqZy
This should have the same net result.
TBH, I'm not 100% convinced we need this export. It is only needed
to send variables to sub-makes. And the only make file reading
FOUND_MOZCONFIG is client.mk. Since the code that evals the
auto-generated make file is always executed in client.mk, we
shouldn't need this export.
All this code is going away soon anyway. So I'm inclined to cargo
cult this just in case.
MozReview-Commit-ID: DqF1BU702A
Currently, client.mk calls `mach environment` to obtain a make file
to be evaluated in the context of client.mk. The reason it is
implemented this way is because client.mk could be an entrypoint to
the build system.
With recent changes that require the use of mach to use client.mk,
we are now guaranteed to have Python code running before client.mk
is invoked. This means we don't need to invoke `mach` from client.mk.
This commit ports the code for generating a client.mk suitable make
file from `mach environment` to the build dispatcher. We now write out
a new .mozconfig-client-mk file in the objdir. client.mk is changed
to cat this file and to include it as a native make file.
The OBJDIR environment variable is also set so client.mk knows where
to read the auto-generated file from.
This commit should be backwards compatible.
Hopefully it is obvious, but this new make file is only temporary.
As soon as the remaining mozconfig logic is moved out of client.mk,
we should be able to simplify down to a single "include" in client.mk.
MozReview-Commit-ID: BEfWo76Z1qA
Instead of evaluating config.guess in client.mk, we evaluate it
in Python. The Python code also looks for CONFIG_GUESS in the
mozconfig. This still happens in client.mk courtesy of evaling
the mozconfig's relevant parts.
MozReview-Commit-ID: 87NmQiB2ccX
We're about to start moving logic from client.mk into Python.
The transition will require doing things like setting
environment variables when we invoke client.mk. Prepare for
this by factoring the "call client.mk" logic into a reusable
function.
The function will obviously be temporary until client.mk is
no more. I'm not sure what the end state will be yet. But surely
having equivalent code in Python is better than having it in
make.
MozReview-Commit-ID: F15DxXSV8V3
This is fairly straightforward to represent as a GENERATED_FILES, though
we have to take some care to construct the outputs tuple correctly. This
script needs to run during export, and unfortunately none of the STL
headers have proper file extensions, so the 'new' header is
special-cased in the recursive make backend to serve as a marker for
running it in the correct tier.
We can't remove the stl-headers file yet because it is still used for
the system header generation.
MozReview-Commit-ID: 3tQTOY0LAsQ
Before, I/O errors writing to stdout/stderr (e.g. due to broken pipe)
would result in handleError() being called and execution would
keep running. This would potentially result in an error message for
every log/line failure being printed to stderr.
We change the behavior so I/O failures are fatal and abort
execution.
We test the new behavior by changing a test to pipe to `head`
directly. Since `head` exits once it has seen sufficient output,
this results in an EPIPE which now results in immediate program
termination.
MozReview-Commit-ID: 1UecZJ56h4r
We currently print compiler warnings summary then stop the build
output manager, which basically stops resource monitoring.
A future commit will refactor this code a bit and it will be easier
to have the monitor.stop() call directly after the build finishes.
MozReview-Commit-ID: 3uWFWG6ELnH
The goal is to move this lower-level code somewhere where it can be
reused.
The only meaningful code change as part of the move was handling
of the mach context. Before, we could access self._mach_context.
After, we need to pass it as an argument. This also uncovered a bug
where "telemetry_handler" could be None but we called it
unconditionally. We now guard this call with an "if."
MozReview-Commit-ID: FWw4l6H1ayS
In general, we shouldn't have so much business logic in mach_commands.py
files.
I'd like to move more of the low-level "perform a build" logic out of
mach_commands.py to facilitate some future work. That code makes use of
this output processing code. So as the first step in moving the build
code, we move its dependencies.
As part of this, I also cleaned up some random unused imports around
lines that were touched.
No meaningful code changes were performed as part of the code move.
MozReview-Commit-ID: 96mGWUJ7oLb
Currently, marking a logger as a structured logger will require a
subsequent function call in order for the logger to be hooked up
to active handlers. This behavior is not intuitive and makes it
easy to not have handlers for newly-registered loggers. This means
messages may not be logged anywhere.
In addition, we have to manually specify which named loggers to
enable structured logging for. This can be annoying.
We change the behavior of register_structured_logger() to
automatically add existing terminal and json handlers to the
logger being marked as structured.
We also introduce an API to enable structured logging for all
loggers. Existing consumers of registered_structured_logger()
in mozbuild have been updated to use this API. A new consumer
has been added for the `mach configure` command because it should
have been there before.
We stop short of making enable_all_structured_loggers() the default.
This is because various commands interact with the log manager in
ways that will result in duplicate logging of messages and
dropping of structured messages. There is a bit of a rabbit hole
here and addressing it can be done as a follow-up.
MozReview-Commit-ID: 1aU6eJvTSMP
We don't use this attribute outside of this function. Besides, you can
easily obtain a handle on the logger by calling
``logging.getLogger('mach')``.
MozReview-Commit-ID: 41vdn6McowW
This is fairly straightforward to represent as a GENERATED_FILES, though
we have to take some care to construct the outputs tuple correctly. This
script needs to run during export, and unfortunately none of the STL
headers have proper file extensions, so the 'new' header is
special-cased in the recursive make backend to serve as a marker for
running it in the correct tier.
We can't remove the stl-headers file yet because it is still used for
the system header generation.
MozReview-Commit-ID: 3tQTOY0LAsQ
The TestMetadata and TestResolver classes aren't technically part of the build
system. The only connection is that they consume some build system output.
The next patch in this series is going to be merging in a bunch of other test
resolving logic from other parts of the tree. Moving this out first allows us
to keep that extra logic out of mozbuild.
MozReview-Commit-ID: 1eq4SjFVCyW
The default debug console line limit is far too short, especially when
debugging a debug build or a build with some logging turned on. I've never
had any issues with turning off the limit, so hopefully this is fine for
everyone else too.
MozReview-Commit-ID: Krlo7XPEjE0
By default the build console is limited to 500 lines. This can be far too
short, especially when building using `mach build --verbose`. I've run with
this value set to 1,000,000 before without issue, so 10,000 should be fine.
MozReview-Commit-ID: GxBUjW7nOYJ
Previously, client.mk made the decision of whether to perform a PGO
build. This required passing around MOZ_PGO and invoking a separate
make target if this variable was set.
In this commit, we move this logic to Makefile.in. We employ a special
mechanism in rules.mk to override the default make target so
`make` evaluates "profiledbuild" if MOZ_PGO is set. This also
required using an explicit target for $(MAKE) invocations inside
the "profiledbuild" rule to avoid infinite recursion.
MozReview-Commit-ID: 8sHiVspMisM
