Increases WPT wdspec timeouts to more realistic values. Because wdspec
tests interact with the browser from an OOP program, they require more
time to run. Interactive browser tests are also known for generally
being more expensive to run.
25 seconds for the default timeout and 120 seconds for the long timeout
are values picked out of the air and likely needs to be further refined
in the future. It is however the current belief that this moves us in
the right direction.
Further improvements to this approach may involve letting wdspec tests
define timeouts on a per-file or a per-test function level through the
use of pytest-timeouts, but this is purely speculative at this point.
It is the current recommendation to adjust the number of tests and the
runtime duration of the tests in a file according to these new defaults.
MozReview-Commit-ID: 4I3Xz9G6lzv
Certain test types have a need for other defaults than the
wpttest.DEFAULT_TIMEOUT and wpttest.LONG_TIMEOUT values. This patch
changes wptrunner to define default- and long timeouts on a test type
level. This allows a test type to override the default durations defined
in the abstract Test.default_timeout and Test.long_timeout.
Concrete classes, such as ReftestTest and WdspecTest, may override these
class properties.
MozReview-Commit-ID: IS6df5vuIDC
Implement gecko leak checking for web-platform-tests, but do not
enable it yet due to oranges. Allows disabling leak checks for a
specific test using a 'leaks' key in test metadata. This causes the
browser to restart before running the (group of) tests with leak
checking disabled. The feature can be enabled by passing --leak-check
on the command line.
MozReview-Commit-ID: 1kBnJkOaeu8
For gecko builds with --disable-optimize everything is very very slow;
enough that the fixed 60s timeout for the browser to start is
insufficient, and various marionette timeouts may also be hit. To
alleviate this problem either disable timeouts or multiply them by the
timeout multiplier which is generally set to around 3 for debug
builds. This seems sufficient to solve the problem on the hardware
developers actually have.
MozReview-Commit-ID: I3zHJGXlpnd
This change renames the following Marionette preferences:
marionette.defaultPrefs.enabled marionette.enabled
marionette.defaultPrefs.port marionette.port
marionette.force-local marionette.forcelocal
marionette.logging marionette.log.level
The old preference names are still usable, but are henceforth considered
deprecated. They will be removed when Firefox 55 ships.
It also registers these preferences in Firefox so that they are
discoverable through about:config.
This patch also refactors testing/marionette/components/marionette.js.
MozReview-Commit-ID: 1dAMQS2e0og
In order to support leak checking in web-platform-tests, we have to
support restarting the browser before a test starts, which is
complicated by the architecture of wptrunner that puts the TestRunner
and associated Executor in a separate process. In the previous
architecture the Executor process grabbed the test to run from the
queue so it wasn't possible to restart the browser once the test was
known without considerable difficulty.
The primary goal of this refactor is to give the TestRunner access to
the test before it is passed to the executor. However imlementing this
was complicated by the poor structure of the existing code. To make
things more comprehensible the TestRunner class is factored into a
more explicit state machine, with a run() loop that either transitions
the runner into the next state or waits for an external command which
causes such a state transition.
MozReview-Commit-ID: 7cvSkYlCSKe
This avoids a cache collision between directories in the main test set
and vendor-specific tests under some different metadata root.
MozReview-Commit-ID: 5WaEtaLJkVD
Ensure that the webdriver tests can access the web-platform.test
hostnames in the same way as other tests by setting the necessary
Firefox pref as part of the capabilities.
MozReview-Commit-ID: At5ENJY7UXA
This avoids a cache collision between directories in the main test set
and vendor-specific tests under some different metadata root.
MozReview-Commit-ID: 5WaEtaLJkVD
Ensure that the webdriver tests can access the web-platform.test
hostnames in the same way as other tests by setting the necessary
Firefox pref as part of the capabilities.
MozReview-Commit-ID: At5ENJY7UXA
It is required to skip certain WebDriver tests imperatively at runtime,
such as for the Get Window Size and Get Window Position commands, which
are only available at certain configurations.
MozReview-Commit-ID: 9ERYkYm34iI
It is required to skip certain WebDriver tests imperatively at runtime,
such as for the Get Window Size and Get Window Position commands, which
are only available at certain configurations.
MozReview-Commit-ID: 9ERYkYm34iI
Before this, the default of "equal_time" resulted in W7 jobs on Windows
debug builds executing over 6 times slower than the shortest chunk.
Chunking by hash of test directory makes the chunk sizes more consistent.
They aren't as consistent as a full path hashing or random chunking. But
it is an improvement.
MozReview-Commit-ID: 7bUSxY8GhH7
Per discussion in the bug, we want automation to behave like local
development mode with regards to running tests in a directory. That
means we want all tests in a directory to be executed together.
We introduce a directory hash chunker. It is like the HashChunker
except it hashes the directory of the test (not the full path). This
ensures that all tests in the same directory end up in the same
chunk.
Compared to HashChunker, this will likely make slow directories
contribute to higher variance in chunk execution times.
MozReview-Commit-ID: CeV0Gi6NRRp
The built-in hash() function uses the backing memory address for
hashing. This is essentially random. Switch to md5 so input is
consistently hashed across processes.
MozReview-Commit-ID: D52uzttE5hc
./mach web-platform-tests tries to access the tests at
http://web-platform.test/. If an HTTP proxy is configured, it tries to
access it via the proxy, which fails. Disabling the proxy entirely
fixes the problem and should probably have no adverse effects, although
a cleaner solution would be to skip proxies only for that domain (as is
done with localhost).
MozReview-Commit-ID: IkIRLwbH16P
