Bug 1773223 - Make the GN processor an independent script. r=firefox-build-system-reviewers,mjf,ahochheiden
The way the processor works currently is that it relies on two different build backends, one of which produces json files for specific configurations, and the other which produces moz.build files from the aggregate of all those configs. Each of these json files is huge, and we actually don't have enough to support all the platforms we're supposed to be supporting. Adding more files is not enticing. Now that we've made the first step described above work in a single pass on a single machine (as opposed to multiple passes on multiple machines previously), we can actually merge both steps and avoid producing the intermediate json files altogether. This will allow to add more configurations without having to worry about the weight of those files. And because this all doesn't need to depend on having the first step hooked up in the build system, we make the whole an independent script rather than a build backend. Differential Revision: https://phabricator.services.mozilla.com/D149210
This commit is contained in:
Mike Hommey
@@ -11,42 +11,7 @@ we have support in the build system for translating GN configuration
|
||||
files into moz.build files. In most cases these moz.build files will be like any
|
||||
others in the tree (except that they shouldn't be modified by hand), however
|
||||
those updating vendored code or building on platforms not supported by
|
||||
Mozilla automation may need to re-generate these files. This is a two-step
|
||||
process, described below.
|
||||
|
||||
Generating GN configs as JSON
|
||||
=============================
|
||||
|
||||
The first step must take place on a machine with access to the ``GN`` tool, which
|
||||
is specified in a mozconfig with::
|
||||
|
||||
export GN=<path/to/gn>
|
||||
|
||||
With this specified, and the tree configured, run::
|
||||
|
||||
$ ./mach build-backend -b GnConfigGen
|
||||
|
||||
This will run ``gn gen`` on projects found in ``GN_DIRS`` specified in moz.build
|
||||
files, injecting variables from the current build configuration and specifying
|
||||
the result be written to a JSON file in ``$objdir/gn-output``. The file will
|
||||
have a name derived from the arguments passed to ``gn gen``, for instance
|
||||
``x64_False_x64_linux.json``.
|
||||
|
||||
If updating upstream sources or vendoring a new project, this step must be
|
||||
performed for each supported build configuration. If adding support for a
|
||||
specific configuration, the generated configuration may be added to existing
|
||||
configs before re-generating the ``moz.build`` files, which should be found under
|
||||
the ``gn-configs`` directory under the vendored project's top-level directory.
|
||||
|
||||
Generating moz.build files from GN JSON configs
|
||||
===============================================
|
||||
|
||||
Once the relevant JSON configs are present under a project's ``gn-configs``
|
||||
directory, run::
|
||||
|
||||
$ ./mach build-backend -b GnMozbuildWriter
|
||||
|
||||
This will combine the configuration files present into a set of moz.build files
|
||||
that will build the given project. Once the result is verified, the resulting
|
||||
moz.build files should be checked in and should build like any other part of
|
||||
mozilla-central.
|
||||
Mozilla automation may need to re-generate these files. This is a per-project
|
||||
process, described in dom/media/webrtc/third_party_build/gn-configs/README.md for
|
||||
webrtc. As of writing, it is very specific to webrtc, and likely doesn't work as-is
|
||||
for other projects.
|
||||
|
||||
@@ -1,42 +0,0 @@
|
||||
# -*- Mode: python; indent-tabs-mode: nil; tab-width: 40 -*-
|
||||
# vim: set filetype=python:
|
||||
# This Source Code Form is subject to the terms of the Mozilla Public
|
||||
# License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
|
||||
gn_vars = {}
|
||||
|
||||
if CONFIG['MOZ_DEBUG']:
|
||||
gn_vars['is_debug'] = True
|
||||
else:
|
||||
gn_vars['is_debug'] = False
|
||||
|
||||
os = CONFIG['OS_TARGET']
|
||||
|
||||
if CONFIG['MOZ_X11']:
|
||||
gn_vars['use_x11'] = True
|
||||
elif os == "Linux":
|
||||
# Only set use_x11 explicitly to false on Linux for no-x11 variants
|
||||
gn_vars['use_x11'] = False
|
||||
|
||||
flavors = {
|
||||
'WINNT': 'win',
|
||||
'Android': 'android',
|
||||
'Linux': 'linux',
|
||||
'Darwin': 'mac' if CONFIG['MOZ_WIDGET_TOOLKIT'] == 'cocoa' else 'ios',
|
||||
'SunOS': 'solaris',
|
||||
'GNU/kFreeBSD': 'freebsd',
|
||||
'DragonFly': 'dragonfly',
|
||||
'FreeBSD': 'freebsd',
|
||||
'NetBSD': 'netbsd',
|
||||
'OpenBSD': 'openbsd',
|
||||
}
|
||||
gn_vars['target_os'] = flavors.get(os)
|
||||
|
||||
arches = {
|
||||
'x86_64': 'x64',
|
||||
'aarch64': 'arm64',
|
||||
}
|
||||
|
||||
gn_vars['host_cpu'] = arches.get(CONFIG['HOST_CPU_ARCH'], CONFIG['HOST_CPU_ARCH'])
|
||||
gn_vars['target_cpu'] = arches.get(CONFIG['CPU_ARCH'], CONFIG['CPU_ARCH'])
|
||||
@@ -23,6 +23,9 @@ with Files("jsapi/**"):
|
||||
BUG_COMPONENT = ("Core", "WebRTC: Signaling")
|
||||
with Files("sdp/**"):
|
||||
BUG_COMPONENT = ("Core", "WebRTC: Signaling")
|
||||
with Files("third_party_build/**"):
|
||||
BUG_COMPONENT = ("Core", "WebRTC")
|
||||
|
||||
|
||||
EXPORTS += [
|
||||
"CubebDeviceEnumerator.h",
|
||||
@@ -76,10 +79,13 @@ if CONFIG["MOZ_WEBRTC_SIGNALING"]:
|
||||
"jsep",
|
||||
"libwebrtcglue",
|
||||
"sdp",
|
||||
"third_party_build",
|
||||
"transportbridge",
|
||||
"/third_party/libwebrtc",
|
||||
]
|
||||
|
||||
if CONFIG["MOZ_WIDGET_TOOLKIT"] == "gtk":
|
||||
DIRS += ["/third_party/pipewire/libpipewire"]
|
||||
|
||||
# Avoid warnings from third-party code that we can not modify.
|
||||
if CONFIG["CC_TYPE"] == "clang-cl":
|
||||
CXXFLAGS += ["-Wno-invalid-source-encoding"]
|
||||
|
||||
@@ -3,29 +3,14 @@
|
||||
/!\ This is only supported on Linux and macOS. If you are on Windows, you can run
|
||||
the script under [WSL](https://docs.microsoft.com/en-us/windows/wsl/install).
|
||||
|
||||
1. Download a version of the `gn` executable that corresponds to
|
||||
`Thu Nov 19 14:14:00 2020`. In our case, that is version `1889 (8fe02009)`.
|
||||
|
||||
- [Linux](https://chrome-infra-packages.appspot.com/p/gn/gn/linux-amd64/+/bvBFKgehaepiKy_YhFnbiOpF38CK26N2OyE1R1jXof0C)
|
||||
- [macOS](https://chrome-infra-packages.appspot.com/p/gn/gn/mac-amd64/+/nXvMRpyJhLhisAcnRmU5s9UZqovzMAhKAvWjax-swioC)
|
||||
1. The script should be run from the top directory of our firefox tree.
|
||||
|
||||
Find the downloaded `.zip` file, unzip and export the location of the
|
||||
executable:
|
||||
```
|
||||
./mach python python/mozbuild/mozbuild/gn_processor.py dom/media/webrtc/third_party_build/gn-configs/webrtc.json
|
||||
```
|
||||
|
||||
unzip gn-mac-amd64.zip && export GN=`pwd`/gn
|
||||
unzip gn-linux-amd64.zip && export GN=`pwd`/gn
|
||||
|
||||
2. It is time to generate the build files. The script should be run from the
|
||||
top directory of our firefox tree.
|
||||
|
||||
bash ./dom/media/webrtc/third_party_build/gn-configs/generate-gn-build-files.sh
|
||||
|
||||
Debugging the generate script itself may prove useful, and one can do this by
|
||||
setting the DEBUG_GEN environment variable to a non-empty value. This will
|
||||
print everything that the script executes.
|
||||
|
||||
3. Checkin all the generated/modified files and try your build!
|
||||
2. Checkin all the generated/modified files and try your build!
|
||||
|
||||
# Adding new configurations to the build
|
||||
|
||||
Edit the `GnConfigGenBackend` class in `python/mozbuild/mozbuild/gn_processor.py` file.
|
||||
Edit the `main` function in the `python/mozbuild/mozbuild/gn_processor.py` file.
|
||||
|
||||
@@ -1,44 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# This Source Code Form is subject to the terms of the Mozilla Public
|
||||
# License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
|
||||
def main():
|
||||
# Check and retrieve command-line arguments
|
||||
if len(sys.argv) != 2:
|
||||
print(__doc__)
|
||||
sys.exit(1) # Return a non-zero value to indicate abnormal termination
|
||||
filein = sys.argv[1]
|
||||
|
||||
if not os.path.isfile(filein):
|
||||
print("error: {} does not exist".format(filein))
|
||||
sys.exit(1)
|
||||
|
||||
f = open(filein, "r")
|
||||
file = f.read()
|
||||
|
||||
# In practice, almost all of the entries in the cflags section have no affect
|
||||
# on the moz.build output files when running ./mach build-backend -b GnMozbuildWriter
|
||||
# There are few exceptions which do: -msse2, -mavx2, -mfma, -fobjc-arc
|
||||
# However, since we're really concerned about removing differences between development
|
||||
# machines, we only need remove the reference to osx sdk. Removing it doesn't change
|
||||
# the generated moz.build files and makes diffs much easier to see.
|
||||
file = re.sub(
|
||||
r' *"-isysroot",\n *".*/Contents/Developer/Platforms/MacOSX\.platform/Developer/SDKs/'
|
||||
'MacOSX([0-9]+\.[0-9]+)?\.sdk",\n',
|
||||
r"",
|
||||
file,
|
||||
)
|
||||
f.close()
|
||||
|
||||
f = open(filein, "w")
|
||||
f.write(file)
|
||||
f.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,82 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Print an Error message if `set -eE` causes the script to exit due to a failed command
|
||||
trap 'echo "*** ERROR *** $? $LINENO Generation did not complete successfully!"' ERR
|
||||
|
||||
# If DEBUG_GEN is set all commands should be printed as they are executed
|
||||
if [ ! "x$DEBUG_GEN" = "x" ]; then
|
||||
set -x
|
||||
fi
|
||||
|
||||
if [ "x$GN" = "x" ]; then
|
||||
echo "GN is not defined, see README.md"
|
||||
exit
|
||||
fi
|
||||
|
||||
if [ -f $GN ]; then
|
||||
echo "GN is $GN"
|
||||
else
|
||||
echo "Path $GN is not found, see README.md"
|
||||
exit
|
||||
fi
|
||||
|
||||
# After this point:
|
||||
# * eE: All commands should succede.
|
||||
# * u: All variables should be defined before use.
|
||||
# * o pipefail: All stages of all pipes should succede.
|
||||
set -eEuo pipefail
|
||||
|
||||
SYS_NAME=`uname`
|
||||
|
||||
# Check for modified files and abort if present.
|
||||
MODIFIED_FILES=`hg status --modified --added --exclude "**/moz.build" --exclude "dom/media/webrtc/third_party_build/**.json"`
|
||||
if [ "x$MODIFIED_FILES" = "x" ]; then
|
||||
# Completely clean the mercurial checkout before proceeding
|
||||
hg update -C -r .
|
||||
hg purge
|
||||
else
|
||||
echo "There are modified files in the checkout. Cowardly aborting!"
|
||||
echo "$MODIFIED_FILES"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
CONFIG_DIR=dom/media/webrtc/third_party_build/gn-configs
|
||||
echo "CONFIG_DIR is $CONFIG_DIR"
|
||||
|
||||
export MOZ_OBJDIR=$(mktemp -d -p . obj-XXXXXXXXXX)
|
||||
./mach configure
|
||||
if [ ! -d $MOZ_OBJDIR ]; then
|
||||
echo "Expected build output directory $MOZ_OBJDIR is missing"
|
||||
exit 1
|
||||
fi
|
||||
./mach build-backend -b GnConfigGen --verbose | tee $MOZ_OBJDIR/build-backend.log
|
||||
cp $MOZ_OBJDIR/third_party/libwebrtc/gn-output/*.json $CONFIG_DIR
|
||||
|
||||
# run some fixup (mostly removing dev-machine dependent info) on json files
|
||||
for THIS_CONFIG in $CONFIG_DIR/*.json
|
||||
do
|
||||
echo "fixup file: $THIS_CONFIG"
|
||||
./$CONFIG_DIR/fixup_json.py $THIS_CONFIG
|
||||
done
|
||||
|
||||
# The symlinks are no longer needed after generating the .json files.
|
||||
if [ -L third_party/libwebrtc/buildtools ]; then
|
||||
rm third_party/libwebrtc/buildtools
|
||||
else
|
||||
rm -rf third_party/libwebrtc/buildtools
|
||||
fi
|
||||
|
||||
if [ -L third_party/libwebrtc/buildtools ]; then
|
||||
rm third_party/libwebrtc/.git
|
||||
else
|
||||
rm -rf third_party/libwebrtc/.git
|
||||
fi
|
||||
|
||||
# After all the json files are generated they are used to generate moz.build files.
|
||||
echo "Building moz.build files from gn json files"
|
||||
./mach build-backend -b GnMozbuildWriter --verbose
|
||||
|
||||
rm -rf $MOZ_OBJDIR
|
||||
|
||||
echo
|
||||
echo "Done generating gn build files. You should now be able to build with ./mach build"
|
||||
73
dom/media/webrtc/third_party_build/gn-configs/webrtc.json
Normal file
73
dom/media/webrtc/third_party_build/gn-configs/webrtc.json
Normal file
@@ -0,0 +1,73 @@
|
||||
{
|
||||
"target_dir": "third_party/libwebrtc",
|
||||
"gn_target": "//:webrtc",
|
||||
"gn_sandbox_variables": {
|
||||
"COMPILE_FLAGS": {
|
||||
"WARNINGS_AS_ERRORS": []
|
||||
},
|
||||
"FINAL_LIBRARY": "webrtc"
|
||||
},
|
||||
"mozilla_flags": [
|
||||
"-fobjc-arc",
|
||||
"-mavx2",
|
||||
"-mfma",
|
||||
"-mfpu=neon",
|
||||
"-msse2"
|
||||
],
|
||||
"write_mozbuild_variables": {
|
||||
"INCLUDE_TK_CFLAGS_DIRS": [
|
||||
"third_party/libwebrtc/modules/desktop_capture/desktop_capture_generic_gn"
|
||||
]
|
||||
},
|
||||
"non_unified_sources": [
|
||||
"third_party/libwebrtc/api/audio_codecs/opus/audio_encoder_opus_config.cc",
|
||||
"third_party/libwebrtc/audio/channel_send_frame_transformer_delegate.cc",
|
||||
"third_party/libwebrtc/common_audio/vad/vad_core.c",
|
||||
"third_party/libwebrtc/common_audio/vad/webrtc_vad.c",
|
||||
"third_party/libwebrtc/modules/audio_coding/codecs/isac/fix/source/decode_plc.c",
|
||||
"third_party/libwebrtc/modules/audio_coding/codecs/isac/fix/source/lpc_masking_model.c",
|
||||
"third_party/libwebrtc/modules/audio_coding/codecs/isac/fix/source/pitch_filter.c",
|
||||
"third_party/libwebrtc/modules/audio_coding/codecs/isac/fix/source/pitch_filter_c.c",
|
||||
"third_party/libwebrtc/modules/audio_coding/neteq/audio_vector.cc",
|
||||
"third_party/libwebrtc/modules/audio_device/android/audio_manager.cc",
|
||||
"third_party/libwebrtc/modules/audio_device/android/audio_record_jni.cc",
|
||||
"third_party/libwebrtc/modules/audio_device/android/audio_track_jni.cc",
|
||||
"third_party/libwebrtc/modules/audio_device/android/opensles_player.cc",
|
||||
"third_party/libwebrtc/modules/audio_device/linux/audio_device_pulse_linux.cc",
|
||||
"third_party/libwebrtc/modules/audio_device/linux/audio_mixer_manager_pulse_linux.cc",
|
||||
"third_party/libwebrtc/modules/audio_device/win/audio_device_core_win.cc",
|
||||
"third_party/libwebrtc/modules/audio_processing/aecm/aecm_core.cc",
|
||||
"third_party/libwebrtc/modules/audio_processing/aecm/aecm_core_c.cc",
|
||||
"third_party/libwebrtc/modules/audio_processing/aecm/aecm_core_mips.cc",
|
||||
"third_party/libwebrtc/modules/audio_processing/aecm/aecm_core_neon.cc",
|
||||
"third_party/libwebrtc/modules/audio_processing/aecm/echo_control_mobile.cc",
|
||||
"third_party/libwebrtc/modules/audio_processing/echo_control_mobile_impl.cc",
|
||||
"third_party/libwebrtc/modules/audio_processing/echo_detector/normalized_covariance_estimator.cc",
|
||||
"third_party/libwebrtc/modules/audio_processing/gain_control_impl.cc",
|
||||
"third_party/libwebrtc/modules/audio_processing/rms_level.cc",
|
||||
"third_party/libwebrtc/modules/desktop_capture/linux/base_capturer_pipewire.cc",
|
||||
"third_party/libwebrtc/modules/desktop_capture/win/screen_capturer_win_gdi.cc",
|
||||
"third_party/libwebrtc/modules/remote_bitrate_estimator/remote_bitrate_estimator_single_stream.cc",
|
||||
"third_party/libwebrtc/modules/rtp_rtcp/source/flexfec_receiver.cc",
|
||||
"third_party/libwebrtc/modules/rtp_rtcp/source/rtcp_packet/tmmbn.cc",
|
||||
"third_party/libwebrtc/modules/rtp_rtcp/source/rtcp_packet/tmmbr.cc",
|
||||
"third_party/libwebrtc/modules/rtp_rtcp/source/rtcp_sender.cc",
|
||||
"third_party/libwebrtc/modules/rtp_rtcp/source/rtp_format_vp8.cc",
|
||||
"third_party/libwebrtc/modules/rtp_rtcp/source/rtp_rtcp_impl2.cc",
|
||||
"third_party/libwebrtc/modules/rtp_rtcp/source/rtp_sender_egress.cc",
|
||||
"third_party/libwebrtc/modules/rtp_rtcp/source/rtp_sender_video.cc",
|
||||
"third_party/libwebrtc/modules/rtp_rtcp/source/ulpfec_generator.cc",
|
||||
"third_party/libwebrtc/modules/rtp_rtcp/source/video_rtp_depacketizer_vp9.cc",
|
||||
"third_party/libwebrtc/modules/third_party/g722/g722_encode.c",
|
||||
"third_party/libwebrtc/modules/video_capture/windows/device_info_ds.cc",
|
||||
"third_party/libwebrtc/modules/video_capture/windows/help_functions_ds.cc",
|
||||
"third_party/libwebrtc/modules/video_capture/windows/sink_filter_ds.cc",
|
||||
"third_party/libwebrtc/modules/video_coding/codecs/vp8/screenshare_layers.cc",
|
||||
"third_party/libwebrtc/third_party/abseil-cpp/absl/strings/numbers.cc",
|
||||
"third_party/libwebrtc/third_party/abseil-cpp/absl/synchronization/blocking_counter.cc",
|
||||
"third_party/libwebrtc/third_party/abseil-cpp/absl/time/internal/cctz/src/time_zone_posix.cc",
|
||||
"third_party/libwebrtc/third_party/abseil-cpp/absl/time/time.cc",
|
||||
"third_party/libwebrtc/video/rtp_video_stream_receiver2.cc",
|
||||
"third_party/libwebrtc/video/video_receive_stream.cc"
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,90 +0,0 @@
|
||||
# -*- Mode: python; indent-tabs-mode: nil; tab-width: 40 -*-
|
||||
# vim: set filetype=python:
|
||||
# This Source Code Form is subject to the terms of the Mozilla Public
|
||||
# License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
|
||||
with Files("**"):
|
||||
BUG_COMPONENT = ("Core", "WebRTC")
|
||||
|
||||
include("/build/gn.mozbuild")
|
||||
|
||||
webrtc_non_unified_sources = [
|
||||
"../../../../third_party/libwebrtc/api/audio_codecs/opus/audio_encoder_opus_config.cc",
|
||||
"../../../../third_party/libwebrtc/audio/channel_send_frame_transformer_delegate.cc",
|
||||
"../../../../third_party/libwebrtc/common_audio/vad/vad_core.c",
|
||||
"../../../../third_party/libwebrtc/common_audio/vad/webrtc_vad.c",
|
||||
"../../../../third_party/libwebrtc/modules/audio_coding/codecs/isac/fix/source/decode_plc.c",
|
||||
"../../../../third_party/libwebrtc/modules/audio_coding/codecs/isac/fix/source/lpc_masking_model.c",
|
||||
"../../../../third_party/libwebrtc/modules/audio_coding/codecs/isac/fix/source/pitch_filter.c",
|
||||
"../../../../third_party/libwebrtc/modules/audio_coding/codecs/isac/fix/source/pitch_filter_c.c",
|
||||
"../../../../third_party/libwebrtc/modules/audio_coding/neteq/audio_vector.cc",
|
||||
"../../../../third_party/libwebrtc/modules/audio_device/android/audio_manager.cc",
|
||||
"../../../../third_party/libwebrtc/modules/audio_device/android/audio_record_jni.cc",
|
||||
"../../../../third_party/libwebrtc/modules/audio_device/android/audio_track_jni.cc",
|
||||
"../../../../third_party/libwebrtc/modules/audio_device/android/opensles_player.cc",
|
||||
"../../../../third_party/libwebrtc/modules/audio_device/linux/audio_device_pulse_linux.cc",
|
||||
"../../../../third_party/libwebrtc/modules/audio_device/linux/audio_mixer_manager_pulse_linux.cc",
|
||||
"../../../../third_party/libwebrtc/modules/audio_device/win/audio_device_core_win.cc",
|
||||
"../../../../third_party/libwebrtc/modules/audio_processing/aecm/aecm_core.cc",
|
||||
"../../../../third_party/libwebrtc/modules/audio_processing/aecm/aecm_core_c.cc",
|
||||
"../../../../third_party/libwebrtc/modules/audio_processing/aecm/aecm_core_mips.cc",
|
||||
"../../../../third_party/libwebrtc/modules/audio_processing/aecm/aecm_core_neon.cc",
|
||||
"../../../../third_party/libwebrtc/modules/audio_processing/aecm/echo_control_mobile.cc",
|
||||
"../../../../third_party/libwebrtc/modules/audio_processing/echo_control_mobile_impl.cc",
|
||||
"../../../../third_party/libwebrtc/modules/audio_processing/echo_detector/normalized_covariance_estimator.cc",
|
||||
"../../../../third_party/libwebrtc/modules/audio_processing/gain_control_impl.cc",
|
||||
"../../../../third_party/libwebrtc/modules/audio_processing/rms_level.cc",
|
||||
"../../../../third_party/libwebrtc/modules/desktop_capture/linux/base_capturer_pipewire.cc",
|
||||
"../../../../third_party/libwebrtc/modules/desktop_capture/win/screen_capturer_win_gdi.cc",
|
||||
"../../../../third_party/libwebrtc/modules/remote_bitrate_estimator/remote_bitrate_estimator_single_stream.cc",
|
||||
"../../../../third_party/libwebrtc/modules/rtp_rtcp/source/flexfec_receiver.cc",
|
||||
"../../../../third_party/libwebrtc/modules/rtp_rtcp/source/rtcp_packet/tmmbn.cc",
|
||||
"../../../../third_party/libwebrtc/modules/rtp_rtcp/source/rtcp_packet/tmmbr.cc",
|
||||
"../../../../third_party/libwebrtc/modules/rtp_rtcp/source/rtcp_sender.cc",
|
||||
"../../../../third_party/libwebrtc/modules/rtp_rtcp/source/rtp_format_vp8.cc",
|
||||
"../../../../third_party/libwebrtc/modules/rtp_rtcp/source/rtp_rtcp_impl2.cc",
|
||||
"../../../../third_party/libwebrtc/modules/rtp_rtcp/source/rtp_sender_egress.cc",
|
||||
"../../../../third_party/libwebrtc/modules/rtp_rtcp/source/rtp_sender_video.cc",
|
||||
"../../../../third_party/libwebrtc/modules/rtp_rtcp/source/ulpfec_generator.cc",
|
||||
"../../../../third_party/libwebrtc/modules/rtp_rtcp/source/video_rtp_depacketizer_vp9.cc",
|
||||
"../../../../third_party/libwebrtc/modules/third_party/g722/g722_encode.c",
|
||||
"../../../../third_party/libwebrtc/modules/video_capture/windows/device_info_ds.cc",
|
||||
"../../../../third_party/libwebrtc/modules/video_capture/windows/help_functions_ds.cc",
|
||||
"../../../../third_party/libwebrtc/modules/video_capture/windows/sink_filter_ds.cc",
|
||||
"../../../../third_party/libwebrtc/modules/video_coding/codecs/vp8/screenshare_layers.cc",
|
||||
"../../../../third_party/libwebrtc/third_party/abseil-cpp/absl/strings/numbers.cc",
|
||||
"../../../../third_party/libwebrtc/third_party/abseil-cpp/absl/synchronization/blocking_counter.cc",
|
||||
"../../../../third_party/libwebrtc/third_party/abseil-cpp/absl/time/internal/cctz/src/time_zone_posix.cc",
|
||||
"../../../../third_party/libwebrtc/third_party/abseil-cpp/absl/time/time.cc",
|
||||
"../../../../third_party/libwebrtc/video/rtp_video_stream_receiver2.cc",
|
||||
"../../../../third_party/libwebrtc/video/video_receive_stream.cc",
|
||||
]
|
||||
|
||||
if CONFIG["MOZ_WIDGET_TOOLKIT"] == "gtk":
|
||||
DIRS += ["../../../../third_party/pipewire/libpipewire"]
|
||||
|
||||
GN_DIRS += ["../../../../third_party/libwebrtc"]
|
||||
|
||||
gn_vars_copy = gn_vars.copy()
|
||||
|
||||
GN_DIRS["../../../../third_party/libwebrtc"].variables = gn_vars_copy
|
||||
GN_DIRS["../../../../third_party/libwebrtc"].mozilla_flags = [
|
||||
"-fobjc-arc",
|
||||
"-mavx2",
|
||||
"-mfma",
|
||||
"-mfpu=neon",
|
||||
"-msse2",
|
||||
]
|
||||
GN_DIRS["../../../../third_party/libwebrtc"].gn_target = "//:webrtc"
|
||||
# We allow warnings for third-party code that can be updated from upstream.
|
||||
GN_DIRS["../../../../third_party/libwebrtc"].sandbox_vars["COMPILE_FLAGS"] = {
|
||||
"WARNINGS_AS_ERRORS": []
|
||||
}
|
||||
GN_DIRS["../../../../third_party/libwebrtc"].sandbox_vars["FINAL_LIBRARY"] = "webrtc"
|
||||
GN_DIRS[
|
||||
"../../../../third_party/libwebrtc"
|
||||
].non_unified_sources += webrtc_non_unified_sources
|
||||
GN_DIRS["../../../../third_party/libwebrtc"].write_mozbuild_vars[
|
||||
"INCLUDE_TK_CFLAGS_DIRS"
|
||||
] = ["third_party/libwebrtc/modules/desktop_capture/desktop_capture_generic_gn"]
|
||||
@@ -11,8 +11,6 @@ backends = {
|
||||
"CppEclipse": "mozbuild.backend.cpp_eclipse",
|
||||
"FasterMake": "mozbuild.backend.fastermake",
|
||||
"FasterMake+RecursiveMake": None,
|
||||
"GnConfigGen": "mozbuild.gn_processor",
|
||||
"GnMozbuildWriter": "mozbuild.gn_processor",
|
||||
"RecursiveMake": "mozbuild.backend.recursivemake",
|
||||
"StaticAnalysis": "mozbuild.backend.static_analysis",
|
||||
"TestManifest": "mozbuild.backend.test_manifest",
|
||||
|
||||
@@ -31,7 +31,6 @@ from mozbuild.frontend.data import (
|
||||
FinalTargetPreprocessedFiles,
|
||||
FinalTargetFiles,
|
||||
GeneratedFile,
|
||||
GnProjectData,
|
||||
HostLibrary,
|
||||
HostSources,
|
||||
IPDLCollection,
|
||||
@@ -199,11 +198,6 @@ class CommonBackend(BuildBackend):
|
||||
self._handle_generated_sources(objdir_files)
|
||||
return False
|
||||
|
||||
elif isinstance(obj, GnProjectData):
|
||||
# These are only handled by special purpose build backends,
|
||||
# ignore them here.
|
||||
return True
|
||||
|
||||
else:
|
||||
return False
|
||||
|
||||
|
||||
@@ -2367,33 +2367,6 @@ VARIABLES = {
|
||||
(...)
|
||||
""",
|
||||
),
|
||||
"GN_DIRS": (
|
||||
StrictOrderingOnAppendListWithFlagsFactory(
|
||||
{
|
||||
"variables": dict,
|
||||
"sandbox_vars": dict,
|
||||
"non_unified_sources": StrictOrderingOnAppendList,
|
||||
"mozilla_flags": list,
|
||||
"gn_target": six.text_type,
|
||||
"write_mozbuild_vars": dict,
|
||||
}
|
||||
),
|
||||
list,
|
||||
"""List of dirs containing gn files describing targets to build. Attributes:
|
||||
- variables, a dictionary containing variables and values to pass
|
||||
to `gn gen`.
|
||||
- sandbox_vars, a dictionary containing variables and values to
|
||||
pass to the mozbuild processor on top of those derived from gn.
|
||||
- non_unified_sources, a list containing sources files, relative to
|
||||
the current moz.build, that should be excluded from source file
|
||||
unification.
|
||||
- mozilla_flags, a set of flags that if present in the gn config
|
||||
will be mirrored to the resulting mozbuild configuration.
|
||||
- gn_target, the name of the target to build.
|
||||
- write_mozbuild_vars, a dictionary containing variables to control
|
||||
code generation of moz.build files.
|
||||
""",
|
||||
),
|
||||
"SPHINX_TREES": (
|
||||
dict,
|
||||
dict,
|
||||
|
||||
@@ -1372,15 +1372,3 @@ class ChromeManifestEntry(ContextDerived):
|
||||
entry = entry.rebase(mozpath.dirname(manifest_path))
|
||||
# Then add the install_target to the entry base directory.
|
||||
self.entry = entry.move(mozpath.dirname(self.path))
|
||||
|
||||
|
||||
class GnProjectData(ContextDerived):
|
||||
def __init__(self, context, target_dir, gn_dir_attrs, non_unified_sources):
|
||||
ContextDerived.__init__(self, context)
|
||||
self.target_dir = target_dir
|
||||
self.non_unified_sources = non_unified_sources
|
||||
self.gn_input_variables = gn_dir_attrs.variables
|
||||
self.gn_sandbox_variables = gn_dir_attrs.sandbox_vars
|
||||
self.mozilla_flags = gn_dir_attrs.mozilla_flags
|
||||
self.gn_target = gn_dir_attrs.gn_target
|
||||
self.write_mozbuild_variables = gn_dir_attrs.write_mozbuild_vars
|
||||
|
||||
@@ -30,7 +30,6 @@ from .data import (
|
||||
FinalTargetFiles,
|
||||
FinalTargetPreprocessedFiles,
|
||||
GeneratedFile,
|
||||
GnProjectData,
|
||||
ExternalStaticLibrary,
|
||||
ExternalSharedLibrary,
|
||||
HostDefines,
|
||||
@@ -620,26 +619,6 @@ class TreeMetadataEmitter(LoggingMixin):
|
||||
**static_args,
|
||||
)
|
||||
|
||||
def _handle_gn_dirs(self, context):
|
||||
for target_dir in context.get("GN_DIRS", []):
|
||||
context["DIRS"] += [target_dir]
|
||||
gn_dir = context["GN_DIRS"][target_dir]
|
||||
for v in ("variables",):
|
||||
if not getattr(gn_dir, "variables"):
|
||||
raise SandboxValidationError(
|
||||
"Missing value for " 'GN_DIRS["%s"].%s' % (target_dir, v),
|
||||
context,
|
||||
)
|
||||
|
||||
non_unified_sources = set()
|
||||
for s in gn_dir.non_unified_sources:
|
||||
source = SourcePath(context, s)
|
||||
if not os.path.exists(source.full_path):
|
||||
raise SandboxValidationError("Cannot find %s." % source, context)
|
||||
non_unified_sources.add(mozpath.join(context.relsrcdir, s))
|
||||
|
||||
yield GnProjectData(context, target_dir, gn_dir, non_unified_sources)
|
||||
|
||||
def _handle_linkables(self, context, passthru, generated_files):
|
||||
linkables = []
|
||||
host_linkables = []
|
||||
@@ -1207,9 +1186,6 @@ class TreeMetadataEmitter(LoggingMixin):
|
||||
if any(k in context for k in ("FINAL_TARGET", "XPI_NAME", "DIST_SUBDIR")):
|
||||
yield InstallationTarget(context)
|
||||
|
||||
for obj in self._handle_gn_dirs(context):
|
||||
yield obj
|
||||
|
||||
# We always emit a directory traversal descriptor. This is needed by
|
||||
# the recursive make backend.
|
||||
for o in self._emit_directory_traversal_from_context(context):
|
||||
|
||||
@@ -7,17 +7,18 @@ from __future__ import absolute_import, print_function, unicode_literals
|
||||
from collections import defaultdict, deque
|
||||
from copy import deepcopy
|
||||
from pathlib import Path
|
||||
import glob
|
||||
from shutil import which
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import six
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
from mozbuild.backend.base import BuildBackend
|
||||
import mozpack.path as mozpath
|
||||
from mozbuild.bootstrap import bootstrap_toolchain
|
||||
from mozbuild.frontend.sandbox import alphabetical_sorted
|
||||
from mozbuild.frontend.data import GnProjectData
|
||||
from mozbuild.util import mkdir
|
||||
|
||||
|
||||
@@ -157,7 +158,7 @@ def filter_gn_config(path, gn_result, sandbox_vars, input_vars, gn_target):
|
||||
gen_path = (Path(path) / "gen").resolve()
|
||||
# Translates the raw output of gn into just what we'll need to generate a
|
||||
# mozbuild configuration.
|
||||
gn_out = {"targets": {}, "sandbox_vars": sandbox_vars, "gn_gen_args": input_vars}
|
||||
gn_out = {"targets": {}, "sandbox_vars": sandbox_vars}
|
||||
|
||||
cpus = {
|
||||
"arm64": "aarch64",
|
||||
@@ -453,25 +454,23 @@ def write_mozbuild(
|
||||
topsrcdir,
|
||||
srcdir,
|
||||
non_unified_sources,
|
||||
gn_config_files,
|
||||
gn_configs,
|
||||
mozilla_flags,
|
||||
write_mozbuild_variables,
|
||||
):
|
||||
|
||||
all_mozbuild_results = []
|
||||
|
||||
for path in sorted(gn_config_files):
|
||||
with open(path, "r") as fh:
|
||||
gn_config = json.load(fh)
|
||||
mozbuild_attrs = process_gn_config(
|
||||
gn_config,
|
||||
topsrcdir,
|
||||
srcdir,
|
||||
non_unified_sources,
|
||||
gn_config["sandbox_vars"],
|
||||
mozilla_flags,
|
||||
)
|
||||
all_mozbuild_results.append(mozbuild_attrs)
|
||||
for gn_config in gn_configs:
|
||||
mozbuild_attrs = process_gn_config(
|
||||
gn_config,
|
||||
topsrcdir,
|
||||
srcdir,
|
||||
non_unified_sources,
|
||||
gn_config["sandbox_vars"],
|
||||
mozilla_flags,
|
||||
)
|
||||
all_mozbuild_results.append(mozbuild_attrs)
|
||||
|
||||
# Translate {config -> {dirs -> build info}} into
|
||||
# {dirs -> [(config, build_info)]}
|
||||
@@ -582,7 +581,6 @@ def write_mozbuild(
|
||||
|
||||
def generate_gn_config(
|
||||
srcdir,
|
||||
output,
|
||||
gn_binary,
|
||||
input_variables,
|
||||
sandbox_variables,
|
||||
@@ -593,8 +591,8 @@ def generate_gn_config(
|
||||
return str(v).lower()
|
||||
return '"%s"' % v
|
||||
|
||||
gn_input_variables = input_variables.copy()
|
||||
gn_input_variables.update(
|
||||
input_variables = input_variables.copy()
|
||||
input_variables.update(
|
||||
{
|
||||
"concurrent_links": 1,
|
||||
"action_pool_depth": 1,
|
||||
@@ -602,7 +600,7 @@ def generate_gn_config(
|
||||
)
|
||||
|
||||
if input_variables["target_os"] == "win":
|
||||
gn_input_variables.update(
|
||||
input_variables.update(
|
||||
{
|
||||
"visual_studio_path": "/",
|
||||
"visual_studio_version": 2015,
|
||||
@@ -610,7 +608,7 @@ def generate_gn_config(
|
||||
}
|
||||
)
|
||||
if input_variables["target_os"] == "mac":
|
||||
gn_input_variables.update(
|
||||
input_variables.update(
|
||||
{
|
||||
"mac_sdk_path": "/",
|
||||
"enable_wmax_tokens": False,
|
||||
@@ -618,120 +616,85 @@ def generate_gn_config(
|
||||
)
|
||||
|
||||
gn_args = "--args=%s" % " ".join(
|
||||
["%s=%s" % (k, str_for_arg(v)) for k, v in six.iteritems(gn_input_variables)]
|
||||
["%s=%s" % (k, str_for_arg(v)) for k, v in six.iteritems(input_variables)]
|
||||
)
|
||||
# Don't make use_x11 part of the string for openbsd to avoid creating
|
||||
# new json files.
|
||||
gn_arg_string = "_".join(
|
||||
[
|
||||
str(input_variables[k])
|
||||
for k in sorted(input_variables.keys())
|
||||
if k != "use_x11" or input_variables["target_os"] != "openbsd"
|
||||
]
|
||||
)
|
||||
out_dir = mozpath.join(output, "gn-output")
|
||||
gen_args = [gn_binary, "gen", out_dir, gn_args, "--ide=json"]
|
||||
print('Running "%s"' % " ".join(gen_args), file=sys.stderr)
|
||||
subprocess.check_call(gen_args, cwd=srcdir, stderr=subprocess.STDOUT)
|
||||
with tempfile.TemporaryDirectory() as tempdir:
|
||||
gen_args = [gn_binary, "gen", tempdir, gn_args, "--ide=json"]
|
||||
print('Running "%s"' % " ".join(gen_args), file=sys.stderr)
|
||||
subprocess.check_call(gen_args, cwd=srcdir, stderr=subprocess.STDOUT)
|
||||
|
||||
gn_config_file = mozpath.join(out_dir, "project.json")
|
||||
gn_config_file = mozpath.join(tempdir, "project.json")
|
||||
|
||||
with open(gn_config_file, "r") as fh:
|
||||
gn_out = json.load(fh)
|
||||
gn_out = filter_gn_config(
|
||||
out_dir, gn_out, sandbox_variables, input_variables, gn_target
|
||||
with open(gn_config_file, "r") as fh:
|
||||
gn_out = json.load(fh)
|
||||
gn_out = filter_gn_config(
|
||||
tempdir, gn_out, sandbox_variables, input_variables, gn_target
|
||||
)
|
||||
return gn_out
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("config", help="configuration in json format")
|
||||
args = parser.parse_args()
|
||||
|
||||
gn_binary = bootstrap_toolchain("gn/gn") or which("gn")
|
||||
if not gn_binary:
|
||||
raise Exception("The GN program must be present to generate GN configs.")
|
||||
|
||||
with open(args.config, "r") as fh:
|
||||
config = json.load(fh)
|
||||
|
||||
topsrcdir = Path(__file__).parent.parent.parent.parent.resolve()
|
||||
|
||||
vars_set = []
|
||||
for is_debug in (True, False):
|
||||
for target_os in ("android", "linux", "mac", "openbsd", "win"):
|
||||
target_cpus = ["x64"]
|
||||
if target_os in ("android", "linux", "mac", "win"):
|
||||
target_cpus.append("arm64")
|
||||
if target_os == "android":
|
||||
target_cpus.append("arm")
|
||||
if target_os in ("android", "linux", "win"):
|
||||
target_cpus.append("x86")
|
||||
for target_cpu in target_cpus:
|
||||
vars = {
|
||||
"host_cpu": "x64",
|
||||
"is_debug": is_debug,
|
||||
"target_cpu": target_cpu,
|
||||
"target_os": target_os,
|
||||
}
|
||||
if target_os == "linux":
|
||||
for use_x11 in (True, False):
|
||||
vars["use_x11"] = use_x11
|
||||
vars_set.append(vars.copy())
|
||||
else:
|
||||
if target_os == "openbsd":
|
||||
vars["use_x11"] = True
|
||||
vars_set.append(vars)
|
||||
|
||||
gn_configs = []
|
||||
for vars in vars_set:
|
||||
gn_configs.append(
|
||||
generate_gn_config(
|
||||
topsrcdir / config["target_dir"],
|
||||
gn_binary,
|
||||
vars,
|
||||
config["gn_sandbox_variables"],
|
||||
config["gn_target"],
|
||||
)
|
||||
)
|
||||
|
||||
os.remove(gn_config_file)
|
||||
|
||||
gn_out_file = mozpath.join(out_dir, gn_arg_string + ".json")
|
||||
with open(gn_out_file, "w") as fh:
|
||||
json.dump(gn_out, fh, indent=4, sort_keys=True, separators=(",", ": "))
|
||||
print("Wrote gn config to %s" % gn_out_file)
|
||||
print("Writing moz.build files")
|
||||
write_mozbuild(
|
||||
topsrcdir,
|
||||
topsrcdir / config["target_dir"],
|
||||
config["non_unified_sources"],
|
||||
gn_configs,
|
||||
config["mozilla_flags"],
|
||||
config["write_mozbuild_variables"],
|
||||
)
|
||||
|
||||
|
||||
class GnConfigGenBackend(BuildBackend):
|
||||
def consume_object(self, obj):
|
||||
if isinstance(obj, GnProjectData):
|
||||
gn_binary = obj.config.substs.get("GN")
|
||||
if not gn_binary:
|
||||
raise Exception(
|
||||
"The GN program must be present to generate GN configs."
|
||||
)
|
||||
|
||||
vars_set = []
|
||||
for is_debug in (True, False):
|
||||
for target_os in ("android", "linux", "mac", "openbsd", "win"):
|
||||
target_cpus = ["x64"]
|
||||
if target_os in ("android", "linux", "mac", "win"):
|
||||
target_cpus.append("arm64")
|
||||
if target_os == "android":
|
||||
target_cpus.append("arm")
|
||||
if target_os in ("android", "linux", "win"):
|
||||
target_cpus.append("x86")
|
||||
for target_cpu in target_cpus:
|
||||
vars = {
|
||||
"host_cpu": "x64",
|
||||
"is_debug": is_debug,
|
||||
"target_cpu": target_cpu,
|
||||
"target_os": target_os,
|
||||
}
|
||||
if target_os == "linux":
|
||||
for use_x11 in (True, False):
|
||||
vars["use_x11"] = use_x11
|
||||
vars_set.append(vars.copy())
|
||||
else:
|
||||
if target_os == "openbsd":
|
||||
vars["use_x11"] = True
|
||||
vars_set.append(vars)
|
||||
|
||||
for vars in vars_set:
|
||||
generate_gn_config(
|
||||
mozpath.join(obj.srcdir, obj.target_dir),
|
||||
mozpath.join(obj.objdir, obj.target_dir),
|
||||
gn_binary,
|
||||
vars,
|
||||
obj.gn_sandbox_variables,
|
||||
obj.gn_target,
|
||||
)
|
||||
return True
|
||||
|
||||
def consume_finished(self):
|
||||
pass
|
||||
|
||||
|
||||
class GnMozbuildWriterBackend(BuildBackend):
|
||||
def consume_object(self, obj):
|
||||
if isinstance(obj, GnProjectData):
|
||||
gn_config_files = glob.glob(
|
||||
mozpath.join(obj.srcdir, "gn-configs", "*.json")
|
||||
)
|
||||
if not gn_config_files:
|
||||
# Check the objdir for a gn-config in to aide debugging in cases
|
||||
# someone is running both steps on the same machine and want to
|
||||
# sanity check moz.build generation for a particular config.
|
||||
gn_config_files = glob.glob(
|
||||
mozpath.join(obj.objdir, obj.target_dir, "gn-output", "*.json")
|
||||
)
|
||||
if gn_config_files:
|
||||
print(
|
||||
"Writing moz.build files based on the following gn configs: %s"
|
||||
% gn_config_files
|
||||
)
|
||||
write_mozbuild(
|
||||
obj.config.topsrcdir,
|
||||
mozpath.join(obj.srcdir, obj.target_dir),
|
||||
obj.non_unified_sources,
|
||||
gn_config_files,
|
||||
obj.mozilla_flags,
|
||||
obj.write_mozbuild_variables,
|
||||
)
|
||||
else:
|
||||
print(
|
||||
"Ignoring gn project '%s', no config files found in '%s'"
|
||||
% (obj.srcdir, mozpath.join(obj.srcdir, "gn-configs"))
|
||||
)
|
||||
return True
|
||||
|
||||
def consume_finished(self):
|
||||
pass
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
Reference in New Issue
Block a user