Run with:
./mach jsshell-bench ares6
This requires a non-artifact build with ac_add_options --enable-js-shell. Otherwise,
a custom JS shell can be specified with --binary.
MozReview-Commit-ID: C8WAxG9pIkk
This sets verbose=True (used by the mach formatter) and compact=False (used by tbplformatter) if
only a single test is specified with either |mach test| or |mach mochitest|.
This way all TEST_STATUS and log lines will be visible when developers are writing a new test.
MozReview-Commit-ID: 2nuKs9DLnx0
This was originally supposed to be fixed in bug 1421799, but I either forgot to
switch the flag, or accidentally clobbered it while resolving conflicts.
MozReview-Commit-ID: Im8OKbswEf0
The end goal here is to be able to use |mach try fuzzy <path>| with tests that
belong to a subsuite. To do this, we need a unique 'task_regex' value for each
subsuite so that we can map a test path back to a set of tasks.
This removes the TEST_FLAVORS dict (which was mostly just a redefinition of the
data in TEST_SUITES), and instead provides two new private mappings:
<flavor> -> suite definition
(<flavor>, <subsuite>) -> suite definition
To retrieve a suite definition given a flavor/subsuite, consumers can now call
get_suite_definition.
MozReview-Commit-ID: 2pe1v1IHUVy
When 'summary_on_shutdown' is True (which is the case for |mach test| and |mach
mochitest|), BaseSummaryFormatters will save the summary information until the
'shutdown' action is received at the end of the logger's lifetime.
Summary information will no longer be dumped on 'suite_end'.
MozReview-Commit-ID: HKtVr5PxfOy
In order for |mach test| and |mach mochitest| to log an overall summary,
every test harness invocation they make needs to use the same structured
logger (otherwise the affected suite will be missing from the summary).
MozReview-Commit-ID: 8LJw7r8SItk
The code in |mach test| for test resolving, should get merged with the TestResolver
class in moztest.resolve. This way it can be shared with other modules and we'll
have a single canonical place for all our test resolving logic.
MozReview-Commit-ID: IHRXXi5mB4G
The TestMetadata and TestResolver classes aren't technically part of the build
system. The only connection is that they consume some build system output.
The next patch in this series is going to be merging in a bunch of other test
resolving logic from other parts of the tree. Moving this out first allows us
to keep that extra logic out of mozbuild.
MozReview-Commit-ID: 1eq4SjFVCyW
find_paths_and_tags() can instantiate a BuildReader to read moz.build
metadata. We switch `mach test` to call this to avoid some redundant
code.
To facilitate `mach try`'s use case, we add test flavors to the
result of find_paths_and_metadata() and change the function name to
reflect its more generic purpose.
MozReview-Commit-ID: Lcz05W8g2dt
This will make the return value more easily extensible and will help
consumers know what the data structures represent.
MozReview-Commit-ID: DaeYsqfMW37
The AutoTry code got moved around in 27b5181a3c77 and c94bf66df122
(bug 1382775). This broke `mach test` without arguments.
This commit imports the new location and uses the new API. As the
inline TODO states, there is still room for improvement. This will
be done in a separate commit.
MozReview-Commit-ID: AKHfdNnfOhl
This is a pretty bad runner. My goal for now is to get something really
basic working and then improve it over time as more tests get added.
Run tests with:
./mach cramtest path/to/dir
Any extra args will be forwarded to the cram binary, e.g:
./mach cramtest -i path/to/dir
Cram help can be seen with:
./mach python -m cram -- -h
MozReview-Commit-ID: 67jYHfYQjWu
Recent changes in bug 1377216 introduced some new Python imports
to some SpiderMonkey test scripts. These modules likely won't be on
sys.path by default.
To ensure these Python processes always work, this commit changes
them to be executed with the Python binary from the build system's
virtualenv.
In the case of jit_test.py and jstests.py, we switch from relying on
the shebang Python (`/usr/bin/env python`). In other cases, we switch
from mach's python (essentially `which python2.7` or `python`).
Previously, some clients may have been using Python 3 via shebang
discovery. This change will force them to Python 2. This might be
unfortunate. But proper Python 3 support in mach and the build system
is a bit of a mess right now. I'd rather we just consistently use
the virtualenv Python because it is more well-defined and it
eliminates a class of bugs around the Python environment being
undefined.
MozReview-Commit-ID: 4NHzWisIgDC
The |mach try| implementation isn't really testing specific so figured while doing this refactor
anyway we could take the opportunity to move it to /tools. This renames the "autotry" module to
"syntax" as one of the 'selectors' under the newly created tryselect module.
MozReview-Commit-ID: DmnGU5sKuCk
The -j try syntax specifies a "job" task. Job tasks are different from
unittests in that they typically don't require a build and run from the
source tree. Examples include linters (eslint, flake8, etc), python tests
(mozbase, mochitest, etc) and misc things like doc generation.
Sometimes, developers might only want to run a specific "job" task with a
syntax like "./mach try -j mozbase".
This means a few assumptions need to be broken in |mach try|. Platforms
and unittests should no longer be required if -j was specified. Most of
the changes in this patch revolve around dealing with those broken
assumptions.
MozReview-Commit-ID: O0y6V2Wwej
sutagent is no longer built or used; devicemanagerSUT is completely
unused. After this change, devicemanagerADB is the only implementation of
devicemanager, and test harness options like --dm_trans are eliminated.
Tri-state flags: I hate them. I don't want to default "--artifact" in
the autotry parser, since not all consumers of that code have a
mozconfig. However, we also can't overwrite the `artifact` value from
a "--no-artifact" flag without knowing if we're changing anything.
Therefore, we add a new "--no-artifact" flag and manage the tri-state
ourselves.
MozReview-Commit-ID: 19gl3mM4YO0
The test-info command provides a report of test results for a named
test on a set of branches over a time period. Test results include
a pass/fail summary, average/min/max duration, and open bugs.
Branches and time period can be modified with optional parameters.
This causes consumers managing defaults themselves to fail to find a default
subsuite for tests, because the manifestparser will have provided a blank
default value by the time they incorporate defaults into a test definition.
This patch removes the provided defaults and updates a number of places assuming
the 'subsuite' field is always present.
MozReview-Commit-ID: 1jPy52VmEPr
This causes consumers managing defaults themselves to fail to find a default
subsuite for tests, because the manifestparser will have provided a blank
default value by the time they incorporate defaults into a test definition.
This patch removes the provided defaults and updates a number of places assuming
the 'subsuite' field is always present.
MozReview-Commit-ID: 1jPy52VmEPr
