The script behind GENERATED_FILES currently opens output files in text mode,
which means that they wind up with CRLF line endings on Windows. With
switching updater.ini to use LOCALIZED_GENERATED_FILES, this means that it
will wind up with different line endings than it currently has.
Changing this to use binary format means that we'll have LF line endings
everywhere, which shouldn't harm anything on Windows as most of our generated
files are source files anyway.
MozReview-Commit-ID: 7rTUDtVGL82
The moz.build files that specify USE_YASM = True will get the value of
AS_DASH_C_FLAG and AS overwritten in PassthruVariables. We can save
those in the BackendTupfile and use them in place of the configure or
default settings as appropriate. This enables compilation of those .s
files that are built with yasm.
MozReview-Commit-ID: J66q8nKQ0an
This accounts for LOCAL_INCLUDES in the moz.build files, as well as the
default INCLUDES specified in config.mk that are used for host
compilation. Since some of the HOST_CFLAGS were also used for linking,
those flags are split off into HOST_C{XX}_LDFLAGS so that
the linker-only flags can be placed in those variables.
MozReview-Commit-ID: J1LxIZVeFJ
The filenames that these objects generate are passed into the _handle_*
methods instead of with a Sources object, so they need to be added to
the BackendTupfile's list of sources separately.
MozReview-Commit-ID: GoqhiJ3Ismm
Both SFLAGS and ASFLAGS are used to compile assembly, but SFLAGS include
DEFINES and LOCAL_INCLUDES whereas ASFLAGS do not. It seems easiest to
just separate them into two different ComputedFlags values so that the
backend can distinguish between the two types.
MozReview-Commit-ID: Bkm3621ImJG
In most cases, relobjdir is the same as relativedir. However they are
different for some objects, notably with nss gyp handling. We need to
output files in relobjdir, so use that when getting the BackendTupfile.
This puts generated files like certdata.c in the directory that
moz.build is expecting.
MozReview-Commit-ID: DG29OulOKAz
The srcdir does not necessarily directly correspond to the objdir, so it
doesn't make sense to tie them together in BackendTupfile. Since the
srcdir was only used for the IPDL sources, we can just replace that
usage with a local variable.
MozReview-Commit-ID: By0N30VTKhh
This change makes the recursive make backend emit slightly different rules
when handling LOCALIZED_GENERATED_FILES vs. GENERATED_FILES.
First, localized file generation is always done in the libs tier.
Second, inputs are allowed to be locale-dependent, which is determined by
the path starting with `en-US/`. These inputs will be run through MERGE_FILE
to determine the actual file path to pass to the script.
Third, the file_generate action now accepts a `--locale` option, and it
gets passed the value of `$(AB_CD)` when generating localized files. If this
option is passed it is also passed as a keyword argument `locale` to the
generation function.
Fourth, the make rules for localized files include an additional dependency
on FORCE when running l10n repacks, so that the targets will always be
rebuilt in that situation.
MozReview-Commit-ID: BfgR8MxxJXZ
LOCALIZED_FILES and LOCALIZED_GENERATED_FILES are analogs of FINAL_TARGET_FILES
and GENERATED_FILES, but they receive special handling in the recursive
make backend so that l10n repacks work properly. To this end, we support
using the output of LOCALIZED_GENERATED_FILES in LOCALIZED_FILES, but not
mixing localized with non-localized targets.
MozReview-Commit-ID: GCJAUfUG8OZ
This change adds LOCALIZED_GENERATED_FILES, which emits GeneratedFile objects
just like GENERATED_FILES. It also adds a `localized` field to GeneratedFile
which will be `True` for objects emitted from LOCALIZED_GENERATED_FILES.
MozReview-Commit-ID: 3iWGLMkbF2C
This is a new module that will provide a place to store some common
abstractions around the 'blessings' module. The main entrypoint is:
from mozterm import Terminal
term = Terminal()
If blessings is available, this will return a blessings.Terminal()
object. If it isn't available, or something went wrong on import,
this will return a NullTerminal() object, which is a drop-in
replacement that does no formatting.
MozReview-Commit-ID: 6c63svm4tM5
- aarch64 has GNU-free toolchain thus /usr/bin/as is N/A by default
but GNU as is required to build bundled ICU.
- Downstream Python splits sqlite3 C module into a separate package
to reduce optional dependencies but _sqlite3 is used by mozprofile.
MozReview-Commit-ID: Lh2ktcqh16f
I believe all backends will need to know which GeneratedFiles are needed
before compilation can start, so we should make that an attribute of the
object. Each backend can then make its own decision about what to do
with the different types of GeneratedFiles.
MozReview-Commit-ID: ByburRx540b
This makes things slightly more inconvenient (having to set two
environment variables instead of one for the simplest case) until a few
patches down the line, when DMD is statically linked, at which point it
will get down to one environment variable every time.
This makes things slightly more inconvenient (having to set two
environment variables instead of one for the simplest case) until a few
patches down the line, when DMD is statically linked, at which point it
will get down to one environment variable every time.
The Firefox UI tests were packaged wrongly, and as such didn't use
the real path as in tree. This patch fixes that by separating the
packaging logic for the harness, and the tests.
Also it updates the mozharness script to run the Firefox UI tests
command by using the test folder as current working directory. This
will make sure that the relative path to the tests is reported.
It's identical to the location in the tree.
MozReview-Commit-ID: 3YVfCw4RWfV
The initial motivation for this patch, was to prevent command lines that are
too long on Windows. To that end, there is a cap to the number of paths that
can be run per job. For now that cap is set to 50. This will allow for an
average path length of 160 characters, which should be sufficient with room to
spare.
But another big benefit of this patch is that we are running more things in
parallel. Previously, mozlint ran each linter in its own subprocess, but that's
it. If running eslint is 90% of the work, it'll still only get a single
process. This means we are wasting cores as soon as the other linters are
finished.
This patch chunks the number of specified paths such that there will be N*L
jobs where 'N' is the number of cores and 'L' is the number of linters. This
means even when there's a dominant linter, we'll be making better use of our
resources. This isn't perfect of course, as some paths might contain a small
number of files, and some will contain a very large number of files. But it's
a start
A limitation to this approach is that if there are fewer paths specified than
there are cores, we won't schedule enough jobs per linter to use those extra
cores. One idea might be to expand specified directories and individually list
all the paths under the directory. But this has some hairy edge cases that
would be tough to catch. Doing this in a non-hacky way would also require a
medium scale refactor.
So I propose further parallelization efforts be destined for follow-ups.
MozReview-Commit-ID: JRRu13AFaii
Historically we built all our binaries in directories in the objdir, then
symlinked them into dist/bin. Some binaries needed to be copied instead
so that certain relative path lookups work properly, so we resorted to
sprinkling `NSDISTMODE=copy` around Makefiles.
This change makes it so we build PROGRAMs (not any other sort of targets)
directly in dist/bin instead. We could do the same for our other targets
with a little more work.
There were several places in the tree that were copying built binaries to
some other place and needed fixup to match the new location of binaries.
On Windows pdb files are left in the objdir where the program was
originally linked. symbolstore.py needs to locate the pdb file both to
determine whether it should dump symbols for a binary and also to copy
the pdb file into the symbol package. We fix this by simply looking for
the pdb file in the current working directory if it isn't present next
to the binary, which matches how we invoke symbolstore.py.
MozReview-Commit-ID: 8TOD1uTXD5e
Descriptions of the original patches, which I had to squash because they needed to land together:
- move all test files
- move and add shell.js files
- move vars in ecma/shell.js and ecma_3/Date/shell.js
- move funcs in ecma_5/RegExp/shell.js into non262/RegExp/shell.js
- move funcs in js1_5/Expressions/shell.js into non262/expressions/shell.js
- move funcs in js1_8_5/extensions/shell.js into non262/extensions/shell.js
- move funcs in ecma_5/shell.js into non262/shell.js and non262/strict/shell.js
- move funcs from ecma_6/shell.js and ecma_7/shell.js
- move and make browser.js files
- remove unnecessary files
- update script locations from move
- remove redeclarations of vars in shell.js
- save a few more old tests
- delete old test files covered by test262
- fix references to scripts from jit-test
- Move Intl directory
Because the default "fast" variant uses fork() on !windows, death tests
are dangerous, as they themselves say. There are race conditions
involving locks that lead to dead locks in the death test process while
disabling the crash reporter (currently, but this could happen for
different code, even the tested code itself).
See https://bugzilla.mozilla.org/show_bug.cgi?id=1419196#c7 for details.
Using the "threadsafe" variant creates new processes for each death
test. This is notably slower, but can't dead-lock because of some random
lock being held by some random other thread at the moment fork occurred.
This combines and expands the list of directories where compilation
currently works in tup, going backwards through the alphabet. Not all of
these directories actually contain compileable code, but this makes it
obvious which top-level directories are not yet enabled. It is likely
that other directories will compile successfully as well - this is
simply a staging point.
MozReview-Commit-ID: Arsk9Oq5XTV
Flags like -DMOZ_APP_NAME="firefox" need to be sent through the shell as
'-DMOZ_APP_NAME="firefox"', otherwise the double-quotes get eaten and
the string is invalid.
MozReview-Commit-ID: 7QN3VTMAY77
This patch fixes a bug in the FasterMake backend and adds a new
test_fastermake.py file to add a test for the behavior.
The FasterMake backend didn't handle wildcards in file names present in
FINAL_TARGET_FILES properly. For an entry like:
FINAL_TARGET_FILES.foo += ['*.xyz']
It would wind up trying to install the files to `dist/bin/foo/*.xyz/`, a
path with a literal asterisk in it. The code seems to have been written
assuming that wildcards would only be present in directory components of
the path. This change fixes this specific case, although it's possible that
it still doesn't handle all permutations of wildcards properly.
MozReview-Commit-ID: rk2tSyDyIu
This commit adds new moz.build sandbox symbols that are intended to be used
for localized files: LOCALIZED_FILES and LOCALIZED_PP_FILES. They are currently
just do-nothing subclasses of FinalTarget[Preprocessor]Files, but the next
change in this series will add support for them to the recursive make backend.
Because they subclass FinalTarget[Preprocessor]Files, build backends that are
not concerned about localized builds should be able to handle them as if
they were FINAL_TARGET[_PP]_FILES without any additional code.
MozReview-Commit-ID: K0baBZ0F7av
This commit adds support for handling LOCALIZED_FILES and LOCALIZED_PP_FILES
in the recursive make backend. They get special handling because they have
a few special needs:
* They run during the libs tier, so that repacks work.
* The filenames cannot be determined at build-backend generation time,
since repacks run configure once and then repack multiple locales using
the generated backend files, so they are written with to be expanded with
MERGE_FILE by make so that the file gets picked up from the proper locale dir.
Other build backends that aren't trying to handle localized builds will
silently handle these like FINAL_TARGET_FILES, which is fine until we revamp
our l10n repack story.
MozReview-Commit-ID: 2LZhPZNhQ4S
In order to determine if we need to re-run configure, we write
a JSON file representing the evaluated mozconfig. If this JSON
file changes, configure (and config.status for that matter) is
out of data and it is re-executed.
This commit moves the generation of that JSON file to Python.
MozReview-Commit-ID: 636rpSY7gOm
This is a pretty straightforward port of the logic. But we
even go a step further: we delete the file in the objdir if there is
no source mozconfig!
MozReview-Commit-ID: AHFFzy6mXRY
The file is a filtered version of the make file that we previously
started generating for client.mk. Why there is special casing for
UPLOAD_EXTRA_FILES, I'm not sure. This smells fishy and is something
I'd like to take a look at once all code is ported out of client.mk.
The removal of the logic from client.mk meant that we could remove
a bunch of code from client.mk related to loading mozconfig files.
We can now simply include the auto-generated make file directly and
be done with it.
MozReview-Commit-ID: 4M5NElQA7iR
Upcoming commits need to move more logic from client.mk. It will
be easier if we have a list of lines in the mozconfig as a local
variable.
MozReview-Commit-ID: 1wFZTfWLGP9
