This removes about 2/3 of the occurrences of nsXPIDLString in the tree. The
places where nsXPIDLStrings are null-checked are replaced with |rv| checks.
The patch also removes a couple of unused declarations from
nsIStringBundle.idl.
Note that nsStringBundle::GetStringFromNameHelper() was merged into
GetStringFromName(), because they both would have had the same signature.
Most of the names passed to nsIStringBundle::{Get,Format}StringFromUTF8Name
have one of the two following forms:
- a 16-bit C string literal, which is then converted to an 8-bit string in
order for the lookup to occur;
- an 8-bit C string literal converted to a 16-bit string, which is then
converted back to an 8-bit string in order for the lookup to occur.
This patch introduces and uses alternative methods that can take an 8-bit C
string literal, which requires changing some signatures in other methods and
functions. It replaces all C++ uses of the old methods.
The patch also changes the existing {Get,Format}StringFromName() methods so
they take an AUTF8String argument for the name instead of a wstring, because
that's nicer for JS code.
Even though there is a method for C++ code and a different one for JS code,
|binaryname| is used so that the existing method names can be used for the
common case in both languages.
The change reduces the number of NS_ConvertUTF8toUTF16 and
NS_ConvertUTF16toUTF8 conversions while running Speedometer v2 from ~270,000 to
~160,000. (Most of these conversions involved the string
"deprecatedReferrerDirective" in nsCSPParser.cpp.)
Makes initing Places services cheaper, by delaying the connection creation to the first time
it's actually needed.
Same way, delays reading the bookmark roots at the first time they are requested.
Deprecates the concept of lazy observers, since they are no more needed, we can just use addObserver.
Simplifies the startup path: always sends "places-init-complete" (both as a category and a topic) when
the connection starts and adds a "locked" database state when we can't get a working connection.
Makes PlacesCategoriesStarter register for the new category, since it's cheaper than being a bookmarks
observer.
Fixes a couple race conditions in keywords and expiration due to new startup timings.
Removes a test in test_keywords.js that is no more easily feasible, since it'd requires a pre-build
places.sqlite that should be kept up-to-date at every version.
MozReview-Commit-ID: 6ccPUZ651m0
Introduce a new SpinningSynchronousClose API that allows to synchronously close
a connection that executed asynchronous statements, by spinning the events loop.
This is expected to be used rarely in particular cases like database corruption.
It is currently [noscript] since the only consumer is cpp, in the future we can
evaluate removing that, if we find more uses for it.
MozReview-Commit-ID: 7SPqutoF9jJ
Instead of opening and closing the favicons database at every startup to check whether
it's corrupt, just use the ATTACH DATABASE query result to achieve that.
Also avoid replacing places.sqlite in case of non-fatal errors during the connection
initialization.
MozReview-Commit-ID: 4VW3dtoSH07
This function is arguably nicer than calling NS_ProcessNextEvent
manually, is slightly more efficient, and will enable better auditing
for NS_ProcessNextEvent when we do Quantum DOM scheduling changes.
Don't enforce wal checkpoints during Places startup, since that can possibly cause a deadlock that would
make some early synchronous API calls bailout. That would indeed cause unexpected intermittent failures
in tests.
Regardless, forcing checkpoints like this in modern filesystems is unlikely to add much value, since
the probabilities to lose the whole contents of the journal are very low.
Additionally we have better startup handling of invalid databases, so we should be able to recover in any case.
MozReview-Commit-ID: G7nISZkd8s2
journal_size_limit doesn't apply to attached databases, so it must be set apart.
Additionally, the current journal_size_limit setting is wrong it should be bytes.
Finally, increase the wal_autocheckpoint value for performance reasons.
Sqlite by default uses a 4MiB autocheckpoint, but we use synchronous=NORMAL, that is a bit more dataloss risky.
For that reason just use half of the default value.
journal_size_limit is set higher than the autocheckpoint value, so that we won't truncate at
every checkpoint.
MozReview-Commit-ID: 2ZUy2Iwkjjc
This patch moves favicons blobs to a separate database names favicons.sqlite.
The dabatase is then ATTACHED to the main Places connection, so that its tables
can be used as if they were all part of the same database.
The favicons.database contains 3 tables:
1. moz_pages_w_icons
This is the way to join with moz_places, through page_url_hash and page_url.
We are not using the place id to avoid possible mismatches between places.sqlite
and favicons.sqlite. This way the database is "portable" and reusable even
if places.sqlite changes.
2. moz_icons
Contains icons payloads, each payload can either be an SVG or a PNG. These
are the only stored formats, any other format is rescaled and converted to
PNG. ICO files are split into single frames and stored as multiple PNGs.
SVG are distinguishable through width == UINT16_MAX
In future the table will also contain mask-icon color for SVG and average
color for PNGs.
The fixed_icon_url_hash is "fixed" to allow quickly fetch root icons, that
means icons like "domain/favicon.ico" that can also be reused for any page
under that domain.
3. moz_icons_to_pages
This is the relation table between icons and pages.
Each page can have multiple icons, each icon can be used by multiple pages.
There is a FOREIGN_KEY constraint between this (child) table and icons
or pages (parents), so that it's not possible to insert non-existing ids
in this table, and if an entry is removed from a parent table, the relation
will be automatically removed from here.
Note though that removing from the relation table won't remove from the
parent tables.
Since the relations are now many-many, it's no more possible to simply join
places with the icons table and obtain a single icon, thus it's suggested that
consumers go through the "page-icon" protocol.
The migration process from the old favicons table is async and interruptible,
it will be restarted along with the favicons service until the temp preference
places.favicons.convertPayloads is set to true.
MozReview-Commit-ID: CUCoL9smRyt
This patch also reports validation errors for left pane queries on the
server, and removes incorrect mobile query migration code.
MozReview-Commit-ID: FT7kAZJapqE
We have data showing that the Places SQLite database can consume
gigabytes of I/O during Firefox test automation jobs. This is
because a number of tests load pages as rapdily as possible,
effectively stress testing Places and SQLite. As the SQLite
database is committed to, we incur I/O for the WAL journal
and when flushing/synchronizing commits. This can add up to a
lot of overhead, especially on spinning disks.
It is important for Places to run during many tests. But it
isn't necessarily important to run with robust I/O guarantees:
SQLite itself has tests that ensure different journal and
synchronizing modes work as advertised.
This commit introduces a preference that changes the SQLite
journal and synchronization modes to be less robust. We use
an in-memory journal so no I/O is incurred for journal writing.
We disable synchronization during commit so no expensive
file(system) flushing is performed. Because setting this
preference would be dangerous for end users, we only honor the
pref if a scary sounding environment variable is set. Hopefully
that's enough of an obstacle to prevent people from footgunning
themselves.
A preliminary Try run reveals this has the potential to shave
hundreds of megabytes of I/O from various jobs. Although this
commit stops short of changing the configuration in automation
to use the new volatile storage preference.
MozReview-Commit-ID: KCoDVzwkSbg
This aims at speeding up DELETE FROM moz_places like queries.
The primary reason of slowness is the FOR EACH ROW trigger that takes care of updating the moz_hosts table when places are removed.
Unfortunately Sqlite doesn't support FOR EACH STATEMENT triggers, that means the trigger will hit multiple times for pages in the same host.
The patch introduces an additional temp table to accumulate hosts during a delete, then a trigger takes care of updating moz_hosts only once per touched host, rather than once per removed place.
MozReview-Commit-ID: BlJRLQZoC07
The problem is due to sanitization happening too late in the shutdown cycle.
The Sanitizer depends on Places shutdown, that recently moved to async shutdown.
That change caused shutdown to happen completely at profile-before-change, unfortunately
during that phase it's impossible to predict which services are already shutdown.
The patch restores the previous Places shutdown procedure, thus clients are notified
earlier, during profile-change-teardown.
Additional meaningful changes:
* Fixes FX_SANITIZE_TOTAL telemetry to properly count total time taken by sanitize.
* Makes each cleanup operation isolated from other errors to try cleaning up as most as possible.
* In case of multiple sanitization sub steps, each step is isolated by a try/catch, the last seen exception is reported upstream.
* Makes FX_SANITIZE_HISTORY actually measure history, not other random stuff.
* Removes TOPIC_SIMULATE_PLACES_MUST_CLOSE_1 since we can now just use profile-change-teardown for shutdown phase 1.
MozReview-Commit-ID: HroLvbi25IC
The bulk of this commit was generated with a script, executed at the top
level of a typical source code checkout. The only non-machine-generated
part was modifying MFBT's moz.build to reflect the new naming.
CLOSED TREE makes big refactorings like this a piece of cake.
# The main substitution.
find . -name '*.cpp' -o -name '*.cc' -o -name '*.h' -o -name '*.mm' -o -name '*.idl'| \
xargs perl -p -i -e '
s/nsRefPtr\.h/RefPtr\.h/g; # handle includes
s/nsRefPtr ?</RefPtr</g; # handle declarations and variables
'
# Handle a special friend declaration in gfx/layers/AtomicRefCountedWithFinalize.h.
perl -p -i -e 's/::nsRefPtr;/::RefPtr;/' gfx/layers/AtomicRefCountedWithFinalize.h
# Handle nsRefPtr.h itself, a couple places that define constructors
# from nsRefPtr, and code generators specially. We do this here, rather
# than indiscriminantly s/nsRefPtr/RefPtr/, because that would rename
# things like nsRefPtrHashtable.
perl -p -i -e 's/nsRefPtr/RefPtr/g' \
mfbt/nsRefPtr.h \
xpcom/glue/nsCOMPtr.h \
xpcom/base/OwningNonNull.h \
ipc/ipdl/ipdl/lower.py \
ipc/ipdl/ipdl/builtin.py \
dom/bindings/Codegen.py \
python/lldbutils/lldbutils/utils.py
# In our indiscriminate substitution above, we renamed
# nsRefPtrGetterAddRefs, the class behind getter_AddRefs. Fix that up.
find . -name '*.cpp' -o -name '*.h' -o -name '*.idl' | \
xargs perl -p -i -e 's/nsRefPtrGetterAddRefs/RefPtrGetterAddRefs/g'
if [ -d .git ]; then
git mv mfbt/nsRefPtr.h mfbt/RefPtr.h
else
hg mv mfbt/nsRefPtr.h mfbt/RefPtr.h
fi
The goal here is to leave creation stuff mostly for JS, so we can
convert it entirely over to a non-threadsafe cycle-collected version
without breaking any existing C++ users.
I didn't do this for a remaining use in nsGlobalWindow.h to avoid
including nsVariant.h all over the place.
The bulk of this commit was generated by running:
run-clang-tidy.py \
-checks='-*,llvm-namespace-comment' \
-header-filter=^/.../mozilla-central/.* \
-fix
