This makes things slightly more inconvenient (having to set two
environment variables instead of one for the simplest case) until a few
patches down the line, when DMD is statically linked, at which point it
will get down to one environment variable every time.
This makes things slightly more inconvenient (having to set two
environment variables instead of one for the simplest case) until a few
patches down the line, when DMD is statically linked, at which point it
will get down to one environment variable every time.
The Firefox UI tests were packaged wrongly, and as such didn't use
the real path as in tree. This patch fixes that by separating the
packaging logic for the harness, and the tests.
Also it updates the mozharness script to run the Firefox UI tests
command by using the test folder as current working directory. This
will make sure that the relative path to the tests is reported.
It's identical to the location in the tree.
MozReview-Commit-ID: 3YVfCw4RWfV
The initial motivation for this patch, was to prevent command lines that are
too long on Windows. To that end, there is a cap to the number of paths that
can be run per job. For now that cap is set to 50. This will allow for an
average path length of 160 characters, which should be sufficient with room to
spare.
But another big benefit of this patch is that we are running more things in
parallel. Previously, mozlint ran each linter in its own subprocess, but that's
it. If running eslint is 90% of the work, it'll still only get a single
process. This means we are wasting cores as soon as the other linters are
finished.
This patch chunks the number of specified paths such that there will be N*L
jobs where 'N' is the number of cores and 'L' is the number of linters. This
means even when there's a dominant linter, we'll be making better use of our
resources. This isn't perfect of course, as some paths might contain a small
number of files, and some will contain a very large number of files. But it's
a start
A limitation to this approach is that if there are fewer paths specified than
there are cores, we won't schedule enough jobs per linter to use those extra
cores. One idea might be to expand specified directories and individually list
all the paths under the directory. But this has some hairy edge cases that
would be tough to catch. Doing this in a non-hacky way would also require a
medium scale refactor.
So I propose further parallelization efforts be destined for follow-ups.
MozReview-Commit-ID: JRRu13AFaii
Historically we built all our binaries in directories in the objdir, then
symlinked them into dist/bin. Some binaries needed to be copied instead
so that certain relative path lookups work properly, so we resorted to
sprinkling `NSDISTMODE=copy` around Makefiles.
This change makes it so we build PROGRAMs (not any other sort of targets)
directly in dist/bin instead. We could do the same for our other targets
with a little more work.
There were several places in the tree that were copying built binaries to
some other place and needed fixup to match the new location of binaries.
On Windows pdb files are left in the objdir where the program was
originally linked. symbolstore.py needs to locate the pdb file both to
determine whether it should dump symbols for a binary and also to copy
the pdb file into the symbol package. We fix this by simply looking for
the pdb file in the current working directory if it isn't present next
to the binary, which matches how we invoke symbolstore.py.
MozReview-Commit-ID: 8TOD1uTXD5e
Descriptions of the original patches, which I had to squash because they needed to land together:
- move all test files
- move and add shell.js files
- move vars in ecma/shell.js and ecma_3/Date/shell.js
- move funcs in ecma_5/RegExp/shell.js into non262/RegExp/shell.js
- move funcs in js1_5/Expressions/shell.js into non262/expressions/shell.js
- move funcs in js1_8_5/extensions/shell.js into non262/extensions/shell.js
- move funcs in ecma_5/shell.js into non262/shell.js and non262/strict/shell.js
- move funcs from ecma_6/shell.js and ecma_7/shell.js
- move and make browser.js files
- remove unnecessary files
- update script locations from move
- remove redeclarations of vars in shell.js
- save a few more old tests
- delete old test files covered by test262
- fix references to scripts from jit-test
- Move Intl directory
Because the default "fast" variant uses fork() on !windows, death tests
are dangerous, as they themselves say. There are race conditions
involving locks that lead to dead locks in the death test process while
disabling the crash reporter (currently, but this could happen for
different code, even the tested code itself).
See https://bugzilla.mozilla.org/show_bug.cgi?id=1419196#c7 for details.
Using the "threadsafe" variant creates new processes for each death
test. This is notably slower, but can't dead-lock because of some random
lock being held by some random other thread at the moment fork occurred.
This combines and expands the list of directories where compilation
currently works in tup, going backwards through the alphabet. Not all of
these directories actually contain compileable code, but this makes it
obvious which top-level directories are not yet enabled. It is likely
that other directories will compile successfully as well - this is
simply a staging point.
MozReview-Commit-ID: Arsk9Oq5XTV
Flags like -DMOZ_APP_NAME="firefox" need to be sent through the shell as
'-DMOZ_APP_NAME="firefox"', otherwise the double-quotes get eaten and
the string is invalid.
MozReview-Commit-ID: 7QN3VTMAY77
This patch fixes a bug in the FasterMake backend and adds a new
test_fastermake.py file to add a test for the behavior.
The FasterMake backend didn't handle wildcards in file names present in
FINAL_TARGET_FILES properly. For an entry like:
FINAL_TARGET_FILES.foo += ['*.xyz']
It would wind up trying to install the files to `dist/bin/foo/*.xyz/`, a
path with a literal asterisk in it. The code seems to have been written
assuming that wildcards would only be present in directory components of
the path. This change fixes this specific case, although it's possible that
it still doesn't handle all permutations of wildcards properly.
MozReview-Commit-ID: rk2tSyDyIu
This commit adds new moz.build sandbox symbols that are intended to be used
for localized files: LOCALIZED_FILES and LOCALIZED_PP_FILES. They are currently
just do-nothing subclasses of FinalTarget[Preprocessor]Files, but the next
change in this series will add support for them to the recursive make backend.
Because they subclass FinalTarget[Preprocessor]Files, build backends that are
not concerned about localized builds should be able to handle them as if
they were FINAL_TARGET[_PP]_FILES without any additional code.
MozReview-Commit-ID: K0baBZ0F7av
This commit adds support for handling LOCALIZED_FILES and LOCALIZED_PP_FILES
in the recursive make backend. They get special handling because they have
a few special needs:
* They run during the libs tier, so that repacks work.
* The filenames cannot be determined at build-backend generation time,
since repacks run configure once and then repack multiple locales using
the generated backend files, so they are written with to be expanded with
MERGE_FILE by make so that the file gets picked up from the proper locale dir.
Other build backends that aren't trying to handle localized builds will
silently handle these like FINAL_TARGET_FILES, which is fine until we revamp
our l10n repack story.
MozReview-Commit-ID: 2LZhPZNhQ4S
In order to determine if we need to re-run configure, we write
a JSON file representing the evaluated mozconfig. If this JSON
file changes, configure (and config.status for that matter) is
out of data and it is re-executed.
This commit moves the generation of that JSON file to Python.
MozReview-Commit-ID: 636rpSY7gOm
This is a pretty straightforward port of the logic. But we
even go a step further: we delete the file in the objdir if there is
no source mozconfig!
MozReview-Commit-ID: AHFFzy6mXRY
The file is a filtered version of the make file that we previously
started generating for client.mk. Why there is special casing for
UPLOAD_EXTRA_FILES, I'm not sure. This smells fishy and is something
I'd like to take a look at once all code is ported out of client.mk.
The removal of the logic from client.mk meant that we could remove
a bunch of code from client.mk related to loading mozconfig files.
We can now simply include the auto-generated make file directly and
be done with it.
MozReview-Commit-ID: 4M5NElQA7iR
Upcoming commits need to move more logic from client.mk. It will
be easier if we have a list of lines in the mozconfig as a local
variable.
MozReview-Commit-ID: 1wFZTfWLGP9
Compiling C/C++/asm just requires the list of sources from _process_unified_sources() and the
Sources/GeneratedSources objects, along with the ComputedFlags and
PerSourceFlags.
The assembler invocation will need to be tweaked to support yasm as
well, but this works so far for the xpcom directory.
MozReview-Commit-ID: 91BSKV9XPLU
Detecting the upstream repository isn't always possible. Using 'git merge-base'
seems to be a more reliable way of detecting outgoing files.
MozReview-Commit-ID: 3zOdTRIcggw
In addition to help fix this bug, having the base revision will allow us
to download the most likely relevant artifacts from CI. For example, in
|mach try| we can search for a 'full-taskgraph.json' artifact from the
most relevant decision task, even if not updated to the latest central.
The ability to find the base revision will also be needed when merging
tools/tryselect/vcs.py into mozversioncontrol.
MozReview-Commit-ID: 2WRRs4AoXcK
Our toolchain detection logic checks whether we can reuse the target
C (resp. C++) compiler for the host compiler. This is generally only
applicable in the not-cross-compiling case, but we had special logic to
check for clang in the cross-compiling case and accept it, as clang is
able to generate code for multiple architectures from a single compiler
binary.
Our recent switch to clang on Android has exposed a problem in this
logic: we would never check whether the target clang, compiling for the
host, could actually find the host's headers. This was especially
problematic on OS X hosts, where the host clang contains special logic
to grovel inside the XCode installation to find C++ headers. The clang
from the NDK, however, was ignorant of the XCode installation.
Therefore, the NDK clang would happily compile code for the host, even
including C headers for the host, but would be hopelessly lost when it
came to compiling C++ headers during the actual build.
In hopes of mitigating this, we now include a check for a representative
header for C and C++ when checking compilers for each of those
languages. This check will detect such problems as the above, and will
also alert people to potentially misconfigured compilers in other
situations.
We need to modify our test framework to cope with headers being
included, since our mock environment isn't actually equipped with a full
set of compilers and headers.
b6adf66f34c6 (bug 1416052) changed the value for "fh" when this code
is called. It can now be an io.BytesIO. This type enforces that
arguments are bytes and doesn't perform automatic type coercion like
most other parts of Python 2.
self.topobjdir is a unicode. And unicode_literals isn't in effect
in this file. So convert self.topobjdir to bytes to make BytesIO
happy.
MozReview-Commit-ID: LrWTKFp3ZKT
The clobber logic is already written in Python. Now that we
always use mach in front of client.mk, we can check the clobber
state before we execute client.mk.
Since we always check the clobber state, we can remove the
CLOBBER files from various dependencies in client.mk. The
clobberer code should ensure everything is in a good state.
The refactor of the clobber Python code required some changes to
its testing. We drop some support for verifying output strings.
But testing this correctly would require a bit of effort. I don't
think it is worth it.
MozReview-Commit-ID: 69CoImCgtNm
Now that mach is being used to invoke client.mk, we can perform
objdir directory creation there.
Removing the use of mkdir_deps meant that we could remove some
included make files which AFAICT were only used to provide
$(mkdir_deps).
MozReview-Commit-ID: 4ZRToz8xqZy
This should have the same net result.
TBH, I'm not 100% convinced we need this export. It is only needed
to send variables to sub-makes. And the only make file reading
FOUND_MOZCONFIG is client.mk. Since the code that evals the
auto-generated make file is always executed in client.mk, we
shouldn't need this export.
All this code is going away soon anyway. So I'm inclined to cargo
cult this just in case.
MozReview-Commit-ID: DqF1BU702A
Currently, client.mk calls `mach environment` to obtain a make file
to be evaluated in the context of client.mk. The reason it is
implemented this way is because client.mk could be an entrypoint to
the build system.
With recent changes that require the use of mach to use client.mk,
we are now guaranteed to have Python code running before client.mk
is invoked. This means we don't need to invoke `mach` from client.mk.
This commit ports the code for generating a client.mk suitable make
file from `mach environment` to the build dispatcher. We now write out
a new .mozconfig-client-mk file in the objdir. client.mk is changed
to cat this file and to include it as a native make file.
The OBJDIR environment variable is also set so client.mk knows where
to read the auto-generated file from.
This commit should be backwards compatible.
Hopefully it is obvious, but this new make file is only temporary.
As soon as the remaining mozconfig logic is moved out of client.mk,
we should be able to simplify down to a single "include" in client.mk.
MozReview-Commit-ID: BEfWo76Z1qA
Instead of evaluating config.guess in client.mk, we evaluate it
in Python. The Python code also looks for CONFIG_GUESS in the
mozconfig. This still happens in client.mk courtesy of evaling
the mozconfig's relevant parts.
MozReview-Commit-ID: 87NmQiB2ccX
We're about to start moving logic from client.mk into Python.
The transition will require doing things like setting
environment variables when we invoke client.mk. Prepare for
this by factoring the "call client.mk" logic into a reusable
function.
The function will obviously be temporary until client.mk is
no more. I'm not sure what the end state will be yet. But surely
having equivalent code in Python is better than having it in
make.
MozReview-Commit-ID: F15DxXSV8V3
