This adds new test verification steps for mochitest/reftest/xpcshell tests
with MOZ_CHAOSMODE=3 (thread scheduling and network thread scheduling).
Enabling all chaos mode features seems to destabilize test verification
so I am only enabling these features for now.
This adds test verification support to the xpcshell test harness. With --verify,
the specified test(s) are run 20 times, then another 20 times in chaos mode.
Tests are run sequentially. I have some interest in running in parallel also,
but this may not be practical: 1. Under normal circumstances, a test does not
run in parallel with itself, so it is arguably an unrealistic mode of operation;
2. Logging fails if it sees a test start after a test with the same name has completed.
New function updateMozinfo() is broken out of runTests() and the remainder of runTests()
is split into two parts: runTests() prepares tests, then calls new function runTestList() to
actually run them. All changes are structural and no change in behavior is expected.
The complexity of the xpcshell test harness has grown over time. I am reluctant
to make it more complex without first trying to simplify it. Here I consolidate
some of the argument passing between functions in an attempt to simplify some
important interfaces. Changes are strictly structural and should result in no
change in behavior.
This environment variable works on both Windows and Linux for force-disabling
the content sandbox, and now does so on macOS as well.
The xpcshell tests force disable the sandbox because they do things like bind()
sockets, which is not compatible with the content sandbox. This is needed now
because bug 1358223 was force upgrading the sandbox from level 0 (disabled) to
level 1 on beta channel, which caused breakage.
MozReview-Commit-ID: 5DGxtoDLp0C
This patch does a few things:
a) Adds the resources location from the .app directory to the read whitelist
b) When it's a non-packaged build, mach run (and various mach tests) set an environment variable for the repo location which we allow reads from.
r=haik,froydnj
MozReview-Commit-ID: KNvAoUs5Ati
This patch does a few things:
a) Adds the resources location from the .app directory to the read whitelist
b) When it's a non-packaged build, mach run (and various mach tests) set an environment variable for the repo location which we allow reads from.
r=haik,froydnj
MozReview-Commit-ID: KNvAoUs5Ati
Supports creating a windowless browser on Linux without an X server. Most of the
changes are just adding branches to avoid calls in to GTK which calls
into X. Some of the bigger additions were adding a separate headless widget
which implements just enough to render a page. A headless look and
feel were also added since there are many calls into GTK in the platform
specific one.
This patch makes it possible to collect code coverage for xpcshell tests using the linux64-jsdcov build. It also enables the use of a 'coverage' flag to disable tests when they are instrumented with the js debugger for code coverage. Lastly, it uses the 'coverage' flag to disable certain tests.
MozReview-Commit-ID: 97VFkJmlwQn
There are a number of things that were going wrong with XPCShell on windows, this patch address them:
1) We were erroring out in Automation.killAndGetStack(), both failing to produce a minidump and failing
to kill the process. This patch fixes both these issues by using mozcrash instead.
2) Occasionally we were occasionally raising a psutil.NoSuchProcess error when attempting to kill the
process after the test. This appears to be caused by a race condition, but is safe to ignore as if the
process doesn't exist, then we don't need to kill it.
3) Spurious "Unable to remove directory" errors in cleanup. This uses mozfile.remove to remedy this.
MozReview-Commit-ID: 1g5qO8fpM7R
There were two assumptions preventing this output from being logged, both
related to the case a test passes and xpcshell returns 0. The first was
that we would not find crash dumps in this case, and would therefore not
need to log the full output of the test process (in the case xpcshell
returned non-zero or a test failed, we would log this output prior to checking
for crashes). The second was that if a test was eligible to retry, we wouldn't
need to store a test's output at all, because this output would only relate to
a failure that we would consider non-fatal.
The first assumption does not hold because it's possible to fatally assert
at shutdown in tests spawning child processes without causing a test failure
or non-zero exit code.
The second assumption followed from the first, and is violated when the first
is violated, because in this case we would consider a found crash fatal even
when a test was eligible to retry.
This patch reverses these assumptions and logs the full output of a test that
passes but produces crash dumps. It's not clear that the existing code intended
for a crash to ever be considered fatal when a test was eligible to retry, but
to change this criteria now would reduce our effective test coverage by
ignoring crashes that are now considered fatal, so after this patch we continue
to consider this scenario fatal. If it is determined these crashes are related
to these tests running in parallel with other tests, or they are not relevant
for some other reason, these tests should be run sequentially, or this criteria
should be changed.
MozReview-Commit-ID: 2PaFSGx2MVR
This adds the ability to use the command line flag '--jscov-dir-prefix' to collect javascript code coverage from xpcshell tests and output it into the specified directory as a JSON file.
MozReview-Commit-ID: 3MZm73SNChL
A subsequent commit will change where the temporary directory is located
in certain test invocations. To do this, we need to teach the xpcshell
harness to use an alternate temporary directory instead of using the system
default (likely specified from TEMP* environment variables).
MozReview-Commit-ID: IUUlYaLBiEC
I'm not especially happy with this method, but the DMD tests are the only
other tests I can find doing anything like this, and I don't have a better
solution than doing it the same way.
MozReview-Commit-ID: JlclyKoUhRf
I'm not especially happy with this method, but the DMD tests are the only
other tests I can find doing anything like this, and I don't have a better
solution than doing it the same way.
MozReview-Commit-ID: JlclyKoUhRf
Head- and tail entries in manifest files are not mandatory on
local/desktop xpcshell tests, and they should not be on remote/B2G either.
This change makes remotexpcshelltests and runxpcshelltests agree on how
to get head- and tail files.
