Commit Graph

193 Commits

Author SHA1 Message Date
Andrew Halberstadt
ecb053eb1f Bug 1372981 - Limit reftest to 4 attempts at recovering after a crash, r=jmaher
MozReview-Commit-ID: IEfM1b6FTHD
2017-06-14 13:25:59 -04:00
Andrew Halberstadt
bce781d9b8 Bug 1373294 - Fix E305 (two blank lines after method or class) in files enabled by flake8 linter, r=jmaher
This is needed before we can upgrade to flake8 3.3.0, as that version starts flagging these errors.

These files were modified by running:
autopep8 --select E305 --in-place -r <dir>

on the affected directories. I did it one dir at a time and verified the result after each.

MozReview-Commit-ID: FmlsfiKIbtr
2017-06-15 12:10:59 -04:00
Andrew Swan
9b227d3b6d Bug 1369525: Make sure reftests still run when legacy extensions are disabled r=jmaher
This won't be an issue until the 57 release but at that time, the
reftest extensions will only load if the extensions.legacy.enabled
preference is set to true and Cu.isInAutomation, which depends on
another preference, is true.  Set the appropriate prefs here for reftests.

MozReview-Commit-ID: JUWCPTsuJU
2017-06-01 21:18:33 -07:00
Geoff Brown
5d3632889b Bug 1364228 - Increase marionette startup timeout for Linux mochitests; r=ato
Our linux32-debug build is very slow to startup when running mochitests on aws.
Sometimes we see similar behavior on other linux platforms. Intermittently,
in this environment, startup takes longer than the 120 seconds that marionette
waits, resulting in test failures in bug 1261598. Increasing the marionette
startup timeout to 180 seconds appears to effectively avoid these failures.
2017-05-17 10:30:32 -06:00
Shing Lyu
a4a7e11348 Bug 1344991 - Continue reftest after a crash r=dbaron,jmaher
MozReview-Commit-ID: CTUaweql66d
2017-03-09 10:49:20 +08:00
Alex Gaynor
478321d9d3 Bug 1294641 - whitelist reads from the .app directory in the macOS sandbox r=froydnj,haik
This patch does a few things:
a) Adds the resources location from the .app directory to the read whitelist
b) When it's a non-packaged build, mach run (and various mach tests) set an environment variable for the repo location which we allow reads from.

r=haik,froydnj

MozReview-Commit-ID: KNvAoUs5Ati
2017-04-07 14:53:19 -04:00
Sebastian Hengst
155e26d497 Backed out changeset 225683fed1d1 (bug 1294641) for breaking crashtests, reftests and Windows builds. r=backout on a CLOSED TREE 2017-04-17 16:29:52 +02:00
Alex Gaynor
d7af07ecbe Bug 1294641 - whitelist reads from the .app directory in the macOS sandbox r=froydnj,haik
This patch does a few things:
a) Adds the resources location from the .app directory to the read whitelist
b) When it's a non-packaged build, mach run (and various mach tests) set an environment variable for the repo location which we allow reads from.

r=haik,froydnj

MozReview-Commit-ID: KNvAoUs5Ati
2017-04-07 14:53:19 -04:00
Carsten "Tomcat" Book
cd13d12db4 Backed out changeset 6bbf9c0c5dac (bug 1344991) for causing bug 1356002 2017-04-13 09:47:01 +02:00
Shing Lyu
2774d8a2f0 Bug 1344991 - Continue reftest after a crash r=dbaron,jmaher
MozReview-Commit-ID: CTUaweql66d
2017-03-09 10:49:20 +08:00
Geoff Brown
67dfff041e Bug 1354042 - Delete pending crash reports between reftests; r=jmaher
Pending crash reports are stored in UAppData, which is typically
outside of the test profile. Pending crash reports left by one test
may affect future tests, so it is important that they be deleted
between tests. However, since UAppData crash data may be important
to developers running tests outside of automation, the harness cannot
delete them unless requested with the --cleanup-crashes option.
2017-04-07 15:46:08 -06:00
Matt Woodrow
a3a97f3f2a Bug 1300355 - Part 3: Disable reftest canvas cache for win 7. r=jmaher 2017-02-24 12:46:58 +13:00
Deepa
bb63c1ff3d Bug 1280572 - Add layout/tools/reftest to flake8 linter r=ahal
MozReview-Commit-ID: Cqllz9BeV7X
2017-02-15 01:20:56 +05:30
Ehsan Akhgari
eeb8cdd945 Bug 1325215 - Ensure that all AudioContext members that need to participate in cycle collection do so; r=padenot
MozReview-Commit-ID: JIL0nFL4lMW
2017-01-18 10:48:40 +01:00
Andrew Halberstadt
b6864170d2 Bug 1330772 - Remove all b2g related code from reftests, r=jmaher
MozReview-Commit-ID: HY1XbcliTNA
2017-01-12 16:37:37 -05:00
Andrew Halberstadt
ce92d2fd40 Bug 1325148 - Temporarily disable leakchecking in crashtests on linux, r=jmaher
This is being disabled because of the leak in bug 1325215. It slipped in because leak
checking in reftest was temporarily failing to turn the job orange. We are disabling
leak checking so we can land the fix ASAP and prevent further bustage from slipping
through in the mean time. Once bug 1325215 has been fixed, this should be backed out.

MozReview-Commit-ID: 74VGhHvZtIG
2016-12-21 17:06:29 -05:00
Sebastian Hengst
9fd7baed40 Backed out changeset 193d05c4b3fc (bug 1325148) 2016-12-27 15:14:18 +01:00
Andrew Halberstadt
8747d5d649 Bug 1325148 - Temporarily disable leakchecking in crashtests on linux, r=jmaher
This is being disabled because of the leak in bug 1325215. It slipped in because leak
checking in reftest was temporarily failing to turn the job orange. We are disabling
leak checking so we can land the fix ASAP and prevent further bustage from slipping
through in the mean time. Once bug 1325215 has been fixed, this should be backed out.

MozReview-Commit-ID: 74VGhHvZtIG
2016-12-21 17:06:29 -05:00
Masatoshi Kimura
cf907ede28 Bug 1322634 - Add a user stylesheet for reftest. r=jmaher,bz
MozReview-Commit-ID: 184UFd1ssbl
2016-12-15 21:59:16 +09:00
Henrik Skupin
620e1de054 Bug 1320073 - Rename marionette-client to marionette-harness and release version 4.0.0. r=gps,maja_zf,mtseng,SingingTree
MozReview-Commit-ID: L6V3b5h8RWm
2016-12-12 13:05:34 +01:00
Andrew Halberstadt
15edcff113 Bug 1301431 - Change reftest harness to accept Namespace object instead of kwargs, r=armenzg
Reftests hackily create a second argument parser and set defaults on it to get around the fact
that the mach command sends the harness a dict rather than a Namespace object. This is bad because:

1. It's much less hacky to just create a Namespace object directly (rather than making a second parser)
2. Most other mach commands actually *do* return a Namespace object, reftest is the odd one out here.

So this patch makes the reftest mach commands convert to the Namespace object, which allows us to get
rid of the hacks in the reftest harness. This also does some light refactoring of the entry points to
the reftest harness so make it more consistent with mochitest and xpcshell.

MozReview-Commit-ID: 5HMfm64wmK2
2016-09-13 09:26:09 -04:00
Xidorn Quan
5e35977fb2 Bug 1297049 - Include tests with query string and hash string when specifying test path. r=dbaron
MozReview-Commit-ID: 27x0vTs1aDa
2016-08-23 10:32:43 +10:00
Andrew Halberstadt
a5e68e668c Bug 1278890 - Add reftest support to test package mach environment, r=armenzg
This adds reftest support to the test package mach environment. It will allow
developers to easily run reftests after checking out an interactive worker.

MozReview-Commit-ID: fBAbfuG5XQ
2016-07-13 10:57:22 -04:00
Krishna
29315be1bc Bug 1276560 - Package hyphenation-dictionaries separately for reftests. r=sebastian
MozReview-Commit-ID: IRvIqGEfc1m
2016-07-05 01:00:15 +08:00
James Graham
41eda57d72 Bug 1278577 - Use simple strings as reftest test ids, r=ahal
The idea of mozlog ids being tuples unfortunately didn't work so well
with external systems that found the tuple|string datatype difficult to
work with. Convert reftests to use simple ids of the form "<test url>
<comparison> <refurl>" e.g. "about:blank == data:text/html," instead of
tuples.

MozReview-Commit-ID: 18jufbssn4A
2016-06-06 16:28:22 +01:00
Geoff Brown
d905d04224 Bug 1274395 - Delete pending crash reports before running tests; r=jmaher 2016-05-25 14:48:42 -06:00
Gijs Kruitbosch
310d327ef8 Bug 1270144 - use error() rather than warning() in order to ensure this shows up on the treeherder summary, rs=emorley-over-irc
MozReview-Commit-ID: COtfpoU14vg
2016-05-25 15:14:56 +01:00
Gijs Kruitbosch
48db4ef098 Bug 1270144 - fix message from timed out tests to indicate the following crash is caused by us terminating the browser process, r=ahal,RyanVM
MozReview-Commit-ID: Hyr7xkZdbqc
2016-05-10 22:53:17 +01:00
Andrew Halberstadt
5ee1903599 Bug 1265124 - [reftest] Make sure marionette waits long enough when attaching a debugger, r=chmanchester
As of bug 1245092, reftest depends on marionette. Normally marionette client has an internal timeout of around
60 seconds where it waits for the server to become available. But when using debuggers (and especially valgrind),
it can take much much longer than a minute for Firefox to start. This patch adds a couple hidden command line
args to reftest to tweak the marionette timeouts.

MozReview-Commit-ID: 3xF0InBJNEf
2016-04-29 15:41:23 -04:00
Joel Maher
14b85fefb0 Bug 1262954 - Explicitly set the 'layers.acceleration.disabled' pref to true on Windows XP e10s test runs. r=ahal
MozReview-Commit-ID: 2KsBZDNvsdU
2016-04-08 14:46:25 -04:00
Andrew Halberstadt
1deb0ab7a1 Bug 1243083 - Enable e10s by default when running reftests, r=jmaher
MozReview-Commit-ID: NlTGwdQ2nw
2016-04-01 11:00:37 -04:00
Andrew Halberstadt
d126feb632 Bug 1245092 - Install reftest and specialpowers extensions at runtime via AddonManager.installTemporaryAddon, r=jgriffin
MozReview-Commit-ID: 5HSa3nFVTeF
2016-03-09 14:38:13 -05:00
Felipe Gomes
82f9c991ea Bug 1250744 - Disable e10s add-ons blocking on testing profiles. r=jmaher 2016-03-02 13:50:55 -03:00
Andrew Halberstadt
83fd8bbf46 Bug 1034290 - Follow-up fix for reftest error message when application returns non-zero, r=chmanchester
MozReview-Commit-ID: JpHzDSuSQqH
2016-02-08 14:54:23 -05:00
Andrew Halberstadt
94aeaa27a9 Bug 1034290 - Use structured log output for test results in reftest, r=jmaher
Structured logs bring many benefits. We can stop parsing the logs for magic strings, we
can modify the format without breaking things, and we can stream results into systems like
ActiveData. The structured logs originate primarily in reftest.js. StructuredLog.jsm is
used to generate the JSON-based log stream. Finally OutputHandler in the python harness
reads structured output from stdout, and formats it into human readable form.

MozReview-Commit-ID: G3ZLkMRl6p7
2016-02-05 15:44:20 -05:00
Phil Ringnalda
c5159ef2e2 Back out d1779fe421c3 (bug 1034290) for UnicodeDecodeErrors in things run by the reftest harness
CLOSED TREE
2016-02-05 08:18:47 -08:00
Andrew Halberstadt
515a1ec282 Bug 1034290 - Use structured log output for test results in reftest, r=jmaher
Structured logs bring many benefits. We can stop parsing the logs for magic strings, we
can modify the format without breaking things, and we can stream results into systems like
ActiveData. The structured logs originate primarily in reftest.js. StructuredLog.jsm is
used to generate the JSON-based log stream. Finally OutputHandler in the python harness
reads structured output from stdout, and formats it into human readable form.
2015-10-29 15:01:44 -04:00
Benoit Girard
113d42ce34 Bug 1196831 - Add 'run-until-failure' and 'repeat' flags to reftest. r=jmaher 2016-01-15 18:30:23 -05:00
Ehsan Akhgari
6b06fec2ca Bug 1215686 - Disable ASAN leak reporting on reftests for now; r=mccr8 2015-10-19 14:08:55 -04:00
Phil Ringnalda
6be41bc877 Back out changeset 7a8d20f52da6 (bug 1215686) for ASan jsreftest/reftest failures
CLOSED TREE
2015-10-16 20:21:54 -07:00
Ehsan Akhgari
0c93af4196 Bug 1215686 - Disable ASAN leak reporting on reftests for now; r=mccr8 2015-10-16 17:46:38 -04:00
Julian Seward
ad8e3b37e6 Bug 1185244 - Improve mach support for running mochitests on Valgrind. r=jgraham, njn. 2015-09-28 12:01:42 +02:00
L. David Baron
41849d962b Bug 1208220 - Remove test of manifest filename that breaks my workflow. r=jgraham 2015-09-27 02:19:13 -07:00
Wes Kocher
ca09e9719a Backed out changeset feceb41f1c3c (bug 1185244) for breaking running tests locally CLOSED TREE a=backout 2015-09-22 17:08:37 -07:00
Julian Seward
a9dcbc18a5 Bug 1185244 - Improve mach support for running mochitests on Valgrind. r=jgraham, njn. 2015-09-22 12:00:57 +02:00
James Graham
7ecb3677b9 Bug 1198257 - Better support for providing a directory name and discovering reftests under that directory, r=jmaher 2015-09-17 10:50:59 +01:00
James Graham
ea97d39711 Bug 1181516 - Allow reftests to take paths to multiple directories containing tests on the command line, r=jmaher
This makes reftest command line arguments behave more like other test suites,
so we can use a simple unified syntax for e.g. |mach try|. The patch also
reworks the command line argument parsing to use argparse rather than optparse,
and causes mach to reuse the same parser as the suite.
2015-09-17 10:50:55 +01:00
James Graham
ebc214bf03 Bug 1181520 - Remove support for passing in reftest arguments via the command line, r=jmaher
This unifies how reftests are invoked across desktop and
mobile, and paves the way for introducing more complex
datatypes that are unreasonable to express on the
command line.
2015-09-17 10:50:53 +01:00
Nathan Froyd
f8b55ca891 Bug 1196430 - part 7 - teach process_leak_log how to symbolicate leaked object stacks; r=mccr8 2015-08-26 19:41:43 -04:00
Nathan Froyd
0c4198a14a Bug 1196430 - part 6 - move cut-and-paste stack fixer code into mozrunner; r=wlach
This code is cut-and-pasted in several different places around the tree.
Let's put it in a common place.
2015-08-26 19:57:36 -04:00