Bug 1855455 - Vendor SQLite and FTS5 as a run-time loadable extension. r=asuth,glandium

Move SQLite updates to the `mach vendor` system to simplify updates.
This will have to be invoked manually through `./mach vendor`, as updatebot
is not hooked up yet (tracking new versions through Github tags is not reliable).

Build FTS5 statically in the same library as SQLite, but as a run-time
loadable extension.
Other extensions will similarly be added in the future, consumers can then use
mozStorage `loadExtension()` or Rusqlite `load_extension()` to load them.
FTS5 is used as a first example and for unit tests. While it could be enabled
statically, we prefer to have it on-demand and per connection.

Differential Revision: https://phabricator.services.mozilla.com/D191314

storage/moz.build

@@ -106,4 +106,7 @@ LOCAL_INCLUDES += [
     "/third_party/sqlite3/src",
 ]
 
+if CONFIG["MOZ_FOLD_LIBS"]:
+    DEFINES["MOZ_FOLD_LIBS"] = True
+
 CXXFLAGS += CONFIG["SQLITE_CFLAGS"]

third_party/sqlite3/README

@@ -1,39 +0,0 @@
-This directory contains source code to 
-
-    SQLite: An Embeddable SQL Database Engine
-
-To compile the project, first create a directory in which to place
-the build products.  It is recommended, but not required, that the
-build directory be separate from the source directory.  Cd into the
-build directory and then from the build directory run the configure
-script found at the root of the source tree.  Then run "make".
-
-For example:
-
-    tar xzf sqlite.tar.gz    ;#  Unpack the source tree into "sqlite"
-    mkdir bld                ;#  Build will occur in a sibling directory
-    cd bld                   ;#  Change to the build directory
-    ../sqlite/configure      ;#  Run the configure script
-    make                     ;#  Run the makefile.
-    make install             ;#  (Optional) Install the build products
-
-The configure script uses autoconf 2.61 and libtool.  If the configure
-script does not work out for you, there is a generic makefile named
-"Makefile.linux-gcc" in the top directory of the source tree that you
-can copy and edit to suit your needs.  Comments on the generic makefile
-show what changes are needed.
-
-The linux binaries on the website are created using the generic makefile,
-not the configure script.  The windows binaries on the website are created
-using MinGW32 configured as a cross-compiler running under Linux.  For 
-details, see the ./publish.sh script at the top-level of the source tree.
-The developers do not use teh configure script.
-
-SQLite does not require TCL to run, but a TCL installation is required
-by the makefiles.  SQLite contains a lot of generated code and TCL is
-used to do much of that code generation.  The makefile also requires
-AWK.
-
-Contacts:
-
-   http://www.sqlite.org/

third_party/sqlite3/README.MOZILLA

@@ -1,13 +0,0 @@
-This is the SQLite amalgamation.
-Check sqlite3.h for the version number and source id.
-
-See http://www.sqlite.org/ for more info.
-
-We have a mozilla-specific moz.build in src/ (normally no
-moz.build there) that we use to build.
-
-To move to a new version:
-
-Copy the sqlite3.h and sqlite3.c files from the amalgamation of sqlite.
-
-Be sure to update SQLITE_VERSION accordingly in $(topsrcdir)/configure.in.

third_party/sqlite3/ext/lib.symbols

@@ -0,0 +1,5 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+sqlite3_fts5_init

third_party/sqlite3/ext/moz.build

@@ -0,0 +1,26 @@
+# -*- Mode: python; indent-tabs-mode: nil; tab-width: 40 -*-
+# vim: set filetype=python:
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+NoVisibilityFlags()
+
+# We allow warnings for third-party code that can be updated from upstream.
+AllowCompilerWarnings()
+
+if CONFIG["MOZ_FOLD_LIBS"]:
+    # When folding libraries, sqlite is actually in the nss library.
+    FINAL_LIBRARY = "nss"
+else:
+    # The final library is in config/external/sqlite
+    FINAL_LIBRARY = "sqlite"
+
+SOURCES += [
+    "fts5.c",
+]
+
+if CONFIG["OS_TARGET"] == "Linux" or CONFIG["OS_TARGET"] == "Android":
+    OS_LIBS += [
+        "m",
+    ]

third_party/sqlite3/moz.yaml

@@ -0,0 +1,49 @@
+schema: 1
+
+bugzilla:
+  product: Toolkit
+  component: Storage
+
+origin:
+  name: SQLite
+  description: >
+    SQLite is a C-language library that implements a small, fast,
+    self-contained, high-reliability, full-featured, SQL database engine.
+  url: https://www.sqlite.org/
+  release: version-3.43.2 (2023-10-10T12:14:04+00:00).
+  revision: version-3.43.2
+  license: Public Domain
+  license-file: LICENSE.md
+  notes: >
+    We have Mozilla specific moz.build and sqlite.symbols in src/ to build the
+    engine. SQLite extensions with similar build files are in ext/.
+    Since this tracks the latest Github tag, if non-versioning tags are added
+    then the update script will fail, and you may have to pass a specific
+    --revision with the appropriate github tag to mach vendor.
+
+vendoring:
+  url: https://github.com/sqlite/sqlite
+  source-hosting: github
+  tracking: tag
+  vendor-directory: third_party/sqlite3/src
+  skip-vendoring-steps:
+    - hg-add
+    - update-moz-build
+  exclude:
+    - "**"
+  include:
+    - LICENSE.md
+    - VERSION
+  keep:
+    - moz.build
+    - sqlite3.c
+    - sqlite3.h
+    - sqlite.symbols
+
+  update-actions:
+    - action: move-file
+      from: '{vendor_dir}/VERSION'
+      to: '{vendor_dir}/VERSION.txt'
+    - action: run-script
+      script: '{yaml_dir}/vendor.sh'
+      cwd: '{yaml_dir}'

third_party/sqlite3/src/LICENSE.md

@@ -0,0 +1,6 @@
+The author disclaims copyright to this source code.  In place of
+a legal notice, here is a blessing:
+
+  *   May you do good and not evil.
+  *   May you find forgiveness for yourself and forgive others.
+  *   May you share freely, never taking more than you give.

third_party/sqlite3/src/VERSION.txt

@@ -0,0 +1 @@
+3.43.2

third_party/sqlite3/src/moz.build

@@ -19,13 +19,16 @@ else:
     # The final library is in config/external/sqlite
     FINAL_LIBRARY = 'sqlite'
 
+DIRS += [
+    '../ext'
+]
+
 SOURCES += [
     'sqlite3.c',
 ]
 
 # -DSQLITE_SECURE_DELETE=1 will cause SQLITE to 0-fill delete data so we
 # don't have to vacuum to make sure the data is not visible in the file.
-# -DSQLITE_CORE=1 statically links that module into the SQLite library.
 # -DSQLITE_DEFAULT_PAGE_SIZE=32768 and SQLITE_MAX_DEFAULT_PAGE_SIZE=32768
 # increases the page size from 1k, see bug 416330.  It must be kept in sync with
 # the value of PREF_TS_PAGESIZE_DEFAULT in mozStorageService.cpp.  The value can
@@ -33,7 +36,7 @@ SOURCES += [
 # hidden preference.  If that preference is missing or invalid then this value
 # will be used.
 # Note: Be sure to update the configure.in checks when these change!
-for var in ('SQLITE_SECURE_DELETE', 'SQLITE_THREADSAFE', 'SQLITE_CORE',
+for var in ('SQLITE_SECURE_DELETE', 'SQLITE_THREADSAFE',
             'SQLITE_ENABLE_UNLOCK_NOTIFY', 'SQLITE_ENABLE_DBSTAT_VTAB'):
     DEFINES[var] = 1
 

third_party/sqlite3/src/sqlite.symbols

@@ -2,6 +2,7 @@
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
+#include ../ext/lib.symbols
 sqlite3_aggregate_context
 sqlite3_auto_extension
 sqlite3_bind_blob

third_party/sqlite3/vendor.sh

@@ -0,0 +1,77 @@
+#!/bin/bash
+
+# IMPORTANT: use `./mach vendor third_party/sqlite3/moz.yaml`, don't invoke
+#            this script directly.
+
+# Script to download updated versions of SQLite sources and extensions.
+
+set -e
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+cd $DIR
+
+# Retrieve SQLite information from the website script-friendly section.
+
+echo ""
+echo "Retrieving SQLite latest version..."
+download_url="https://www.sqlite.org/download.html"
+# The download page contains a script-friendly comment to extract file versions.
+# PRODUCT,VERSION,RELATIVE-URL,SIZE-IN-BYTES,SHA3-HASH
+# Match on the amalgamation to extract path and version, they are the same for
+# all the files anyway.
+re="PRODUCT,([^,]+),([^,]+)/sqlite-amalgamation-([0-9]+)\.zip"
+DOWNLOAD_PAGE_HTML="`wget -t 3 --retry-connrefused -w 5 --random-wait $download_url -qO-`"
+if [[ $DOWNLOAD_PAGE_HTML =~ $re ]]; then
+  webversion="${BASH_REMATCH[1]}";
+  path="${BASH_REMATCH[2]}";
+  version="${BASH_REMATCH[3]}";
+else
+  echo "Error retrieving SQLite files";
+  exit;
+fi
+
+# Check version matches with the one from Github, otherwise you'll have to point
+# ./mach vendor to a specific revision.
+
+echo ""
+echo "Comparing Github and Website version numbers..."
+gitversion=`cat src/VERSION.txt`
+echo "Website version: $webversion";
+echo "Github version: $gitversion";
+if [ "$webversion" != "$gitversion" ]; then
+  echo 'Versions do not match, try to invoke `./mach vendor` with a specific `--revision <github tag>`)'
+  exit;
+fi
+
+# Retrieve files and update sources.
+
+echo ""
+echo "Retrieving SQLite amalgamation..."
+amalgamation_url="https://www.sqlite.org/$path/sqlite-amalgamation-$version.zip"
+wget -t 3 --retry-connrefused -w 5 --random-wait $amalgamation_url -qO amalgamation.zip
+echo "Unpacking SQLite source files..."
+unzip -p "amalgamation.zip" "sqlite-amalgamation-$version/sqlite3.c" > "src/sqlite3.c"
+unzip -p "amalgamation.zip" "sqlite-amalgamation-$version/sqlite3.h" > "src/sqlite3.h"
+mkdir -p ext
+unzip -p "amalgamation.zip" "sqlite-amalgamation-$version/sqlite3ext.h" > "ext/sqlite3ext.h"
+rm -f "amalgamation.zip"
+
+echo ""
+echo "Retrieving SQLite preprocessed..."
+preprocessed_url="https://www.sqlite.org/$path/sqlite-preprocessed-$version.zip"
+wget -t 3 --retry-connrefused -w 5 --random-wait $preprocessed_url -qO preprocessed.zip
+echo "Unpacking FTS5 extension..."
+unzip -p "preprocessed.zip" "sqlite-preprocessed-$version/fts5.c" > "ext/fts5.c"
+rm -f "preprocessed.zip"
+
+# Retrieve and update other SQLite extensions code.
+
+# If the extension is hosted on Github or other supported platforms, you want
+# to use `mach vendor` for it, rather than manually downloading it here.
+# The same is valid for SQLite owned extensions that don't need preprocessing
+# (e.g. carray.c/h). In general anything that is in sqlite-src archive is also
+# in their official Github repo.
+
+echo ""
+echo "Update complete, please commit and check in your changes."
+echo ""