a3bfaf1f42
Automatic update from web-platform-tests Improve idlharness-ShadowRealm integration By exposing a fetch_spec() on globalThis in testharness-shadowrealm-inner, we no longer need to have IDL tests in separate files for ShadowRealm, as the regular idlharness.js will just run there. Delete all existing idlharness-shadowrealm.window.js files and incorporate them into their neighbouring idlharness.any.js files. It seems sufficient to run the IDL tests only in shadowrealm-in-window; repeating them in every ShadowRealm scope seems like overkill. -- Improve test message in idlharness This gets me every time. If we are testing that an interface is not exposed, we should be clear about that in the test message. -- Don't test namespaces in idlharness if not exposed Instead, if a namespace is not exposed, test that there is no global property of that name, similar to what we do with interfaces. -- wpt-commits: 67880a4eb83ca9aa732eec4b35a1971ff5bf37ff, 524e1e0643c9d42e74f0bec6092227738b605f76, 49980192b50a30cd8bbea9b6baab9ac191eb1eb7 wpt-pr: 49342
testharness.js test suite
The test suite for the testharness.js testing framework.
Executing Tests
Install the following dependencies:
Make sure geckodriver can be found in your PATH.
Currently, the tests should be run with the latest Firefox Nightly. In order to specify the path to Firefox Nightly, use the following command-line option:
tox -- --binary=/path/to/FirefoxNightly
Automated Script
Alternatively, you may run tools/ci/ci_resources_unittest.sh, which only depends on
Python 2. The script will install other dependencies automatically and start tox with
the correct arguments.
Authoring Tests
Test cases are expressed as .html files located within the tests/unit/ and
tests/functional/ sub-directories. Each test should include the
testharness.js library with the following markup:
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
This should be followed by one or more <script> tags that interface with the
testharness.js API in some way. For example:
<script>
test(function() {
1 = 1;
}, 'This test is expected to fail.');
</script>
Unit tests
The "unit test" type allows for concisely testing the expected behavior of assertion methods. These tests may define any number of sub-tests; the acceptance criteria is simply that all tests executed pass.
Functional tests
Thoroughly testing the behavior of the harness itself requires ensuring a number of considerations which cannot be verified with the "unit testing" strategy. These include:
- Ensuring that some tests are not run
- Ensuring conditions that cause test failures
- Ensuring conditions that cause harness errors
Functional tests allow for these details to be verified. Every functional test
must include a summary of the expected results as a JSON string within a
<script> tag with an id of "expected", e.g.:
<script type="text/json" id="expected">
{
"summarized_status": {
"message": null,
"stack": null,
"status_string": "OK"
},
"summarized_tests": [
{
"message": "ReferenceError: invalid assignment left-hand side",
"name": "Sample HTML5 API Tests",
"properties": {},
"stack": "(implementation-defined)",
"status_string": "FAIL"
}
],
"type": "complete"
}
</script>