This patch rewrites the rust-url-capi crate as the mozurl crate, which
provides a threadsafe MozURL object which is compatible with the
previous MozURL class.
Creating a MozURL this way performs a single allocation, which contains
only a rust-url Url object and an atomic refcnt, however it is fully
compatible with the C++ RefPtr type.
This patch also exposes methods for accessing dependent substrings of
the serialized spec, meaning that string copies can be avoided in many
situations when inspecting attributes of the MozURL.
Remove the syn dependency from gkrust-shared, which was added in bug 1373878
as a workaround for this Cargo bug:
https://github.com/rust-lang/cargo/issues/3923
MozReview-Commit-ID: L34J0davEYd
The prefs parser has two significant problems.
- It doesn't separate tokenizing from parsing.
- It is implemented as a loop around a big switch on a "current state"
variable.
As a result, it is hard to understand and modify, slower than it could be, and
in obscure cases (involving comments and whitespace) it fails to parse what
should be valid input.
This patch replaces it with a recursive descent parser (albeit one without any
recursion!) that has separate tokenization. The new parser is easier to
understand and modify, more correct, and has better error messages. It doesn't
do error recovery, but that would be much easier to add than in the old parser.
The new parser also runs about 1.9x faster than the existing parser. (As
measured by parsing greprefs.js's contents from memory 1000 times in
succession, omitting the prefs hash table construction. If the table
construction is included, it's about 1.6x faster.)
The new parser is slightly stricter than the old parser in a few ways.
- Disconcertingly, the old parser allowed arbitrary junk between prefs
(including at the start and end of the prefs file) so long as that junk
didn't include any of the following chars: '/', '#', 'u', 's', 'p'. I.e.
lines like these:
!foo@bar&pref("prefname", true);
ticky_pref("prefname", true); // missing 's' at start
User_pref("prefname", true); // should be 'u' at start
would all be treated the same as this:
pref("prefname", true);
The new parser disallows such junk because it isn't necessary and seems like
an unintentional botch by the old parser.
- The old parser allowed character 0x1a (SUB) between tokens and treated it
like '\n'.
The new parser does not allow this character. SUB was used to indicate
end-of-file (*not* end-of-line) in some old operating systems such as MS-DOS,
but this doesn't seem necessary today.
- The old parser tolerated (with a warning) invalid escape sequences within
string literals -- such as "\q" (not a valid escape) and "\x1" and "\u12"
(both of which have insufficient hex digits) -- accepting them literally.
The new parser does not tolerate invalid escape sequences because it doesn't
seem necessary and would complicate things.
- The old parser tolerated character 0x00 (NUL) within string literals; this is
dangerous because C++ code that manipulates string values with embedded NULs
will almost certainly consider those chars as end-of-string markers.
The new parser treats NUL chars as end-of-file, to avoid this danger and
because it facilitates a significant optimization (described within the
code).
- The old parser allowed integer literals to overflow, silently wrapping them.
The new parser treats integer overflow as a parse error. This seems better,
and it caught existing overflows of places.database.lastMaintenance, in
testing/profiles/prefs_general.js (bug 1424030) and
testing/talos/talos/config.py (bug 1434813).
The first of these changes meant that a couple of existing prefs with ";;" at
the end had to be changed (done in the preceding patch).
The minor increase in strictness shouldn't be a problem for default pref files
such as greprefs.js within the application (which we can modify), nor for
app-written prefs files such as prefs.js. It could affect user-written prefs
files such as user.js; the experience above suggests that integer overflow and
";;" are the most likely problems in practice. In my opinion, the risk here is
acceptable.
The new parser also does a better job of tracking line numbers because it (a)
treats "\r\n" sequences as a single end-of-line marker, and (a) pays attention
to end-of-line sequences within string literals.
Finally, the patch adds thorough tests of both valid and invalid syntax.
MozReview-Commit-ID: JD3beOQl4AJ
The prefs parser has two significant problems.
- It doesn't separate tokenizing from parsing.
- It is implemented as a loop around a big switch on a "current state"
variable.
As a result, it is hard to understand and modify, slower than it could be, and
in obscure cases (involving comments and whitespace) it fails to parse what
should be valid input.
This patch replaces it with a recursive descent parser (albeit one without any
recursion!) that has separate tokenization. The new parser is easier to
understand and modify, more correct, and has better error messages. It doesn't
do error recovery, but that would be much easier to add than in the old parser.
The new parser also runs about 1.9x faster than the existing parser. (As
measured by parsing greprefs.js's contents from memory 1000 times in
succession, omitting the prefs hash table construction. If the table
construction is included, it's about 1.6x faster.)
The new parser is slightly stricter than the old parser in a few ways.
- Disconcertingly, the old parser allowed arbitrary junk between prefs
(including at the start and end of the prefs file) so long as that junk
didn't include any of the following chars: '/', '#', 'u', 's', 'p'. I.e.
lines like these:
!foo@bar&pref("prefname", true);
ticky_pref("prefname", true); // missing 's' at start
User_pref("prefname", true); // should be 'u' at start
would all be treated the same as this:
pref("prefname", true);
The new parser disallows such junk because it isn't necessary and seems like
an unintentional botch by the old parser.
- The old parser allowed character 0x1a (SUB) between tokens and treated it
like '\n'.
The new parser does not allow this character. SUB was used to indicate
end-of-file (*not* end-of-line) in some old operating systems such as MS-DOS,
but this doesn't seem necessary today.
- The old parser tolerated (with a warning) invalid escape sequences within
string literals -- such as "\q" (not a valid escape) and "\x1" and "\u12"
(both of which have insufficient hex digits) -- accepting them literally.
The new parser does not tolerate invalid escape sequences because it doesn't
seem necessary and would complicate things.
- The old parser tolerated character 0x00 (NUL) within string literals; this is
dangerous because C++ code that manipulates string values with embedded NULs
will almost certainly consider those chars as end-of-string markers.
The new parser treats NUL chars as end-of-file, to avoid this danger and
because it facilitates a significant optimization (described within the
code).
- The old parser allowed integer literals to overflow, silently wrapping them.
The new parser treats integer overflow as a parse error. This seems better,
and it caught an existing overflow in testing/profiles/prefs_general.js, for
places.database.lastMaintenance (see bug 1424030).
The first of these changes meant that a couple of existing prefs with ";;" at
the end had to be changed (done in the preceding patch).
The minor increase in strictness shouldn't be a problem for default pref files
such as greprefs.js within the application (which we can modify), nor for
app-written prefs files such as prefs.js. It could affect user-written prefs
files such as user.js; the experience above suggests that ";;" is the most
likely problem in practice. In my opinion, the risk here is acceptable.
The new parser also does a better job of tracking line numbers because it (a)
treats "\r\n" sequences as a single end-of-line marker, and (a) pays attention
to end-of-line sequences within string literals.
Finally, the patch adds thorough tests of both valid and invalid syntax.
MozReview-Commit-ID: 8EYWH7KxGG
* * *
[mq]: win-fix
MozReview-Commit-ID: 91Bxjfghqfw
The easy part of this patch is the addition of the RustTest itself.
The more difficult to understand part of the patch is the changes to all
of our Rust build configuration. We do this due to a bug in cargo:
https://github.com/rust-lang/cargo/issues/3923
where features on dependent crates are not correctly taken into account
when determining whether cached artifacts on disk are valid and whether
they should be evicted from the disk cache. The practical upshot of
this behavior is that, say, running gtests during normal development
when files in libxul are modified will:
* rebuild some Rust dependencies for libxul;
* link libxul;
* rebuild those same Rust dependencies *again* for libxul-gtest, since
we have different features active and therefore the old artifacts look
to be out of date;
* link libxul-gtest.
Needless to say, this is highly annoying and counterproductive behavior.
The "fix" is to ensure that the gkrust-shared crate explicitly depends
on crates and assigns features to them such that the feature sets do not
change between normal builds and testing builds. This is admittedly
fragile, but it is not the first time this has come up, and is probably
not the last.
Crash reporter installs a special Rust panic hook to grab the panic reason.
However, we still want to call the default hook as well, so that we still print
the reason and backtrace to the console.
MozReview-Commit-ID: JlCamBPb51X
For testing purposes it will be useful to be able to trigger crashes in Rust
code. Being able to trigger a panic seems like a good place to start. This
will make it easier to validate improvements in crash reporting.
MozReview-Commit-ID: Bh5rBieLLWW
For testing purposes it will be useful to be able to trigger crashes in Rust
code. Being able to trigger a panic seems like a good place to start. This
will make it easier to validate improvements in crash reporting.
MozReview-Commit-ID: Bh5rBieLLWW
In our current Rust world, we have the following dependency structure:
xul.so --------------------------+
|
xul-gtest.so -+--> xul.a --------+-> gkrust
|
+--> gkrust-gtest
This structure results in link errors with multiply-defined symbols
between gkrust-gtest and gkrust with newer Rust releases when linking
xul-gtest.so. So we have to do something different.
Our new structure is:
xul.so --------------------------+
|
xul-gtest.so -+--> xul.a --------+-> gkrust --+-> gkrust-shared
| |
+--> gkrust-gtest --------------+
and we enforce that a given shared library can only have at most one
Rust library that it depends on. Said Rust library is assumed to
include all significant Rust dependencies of the dependent static
libraries as well. (In the above structure, gkrust is simply a wrapper
around gkrust-shared, so gkrust-gtest doesn't have to include gkrust as
a dependency.)
