This patch allows us to enable Python 3 tests and skip any tests that fail so that we can work on adding support for Python 3 without risking regressing any existing support. It will also eventually allow us to skip tests from running against Python 2, when we decide to drop support for it. To skip a test against Python 3, add "skip-if = python == 3" to the [DEFAULT] or test file section of a manifest file.
MozReview-Commit-ID: KYzjW6PQw2Q
This patch allows executing |mach python-test| against Python 3 by specifying the optional |--three| command line option. When this option is present, pipenv will be used to manage a virtual environment using Python 3 and attempt to run the tests. When it is not present, pipenv will not be used, and everything will work as it did before this patch.
My original plan was to use pipenv regardless of the target version of Python, however I encountered several issues running some of the tests against our Python packages. Once all tests have been patched to run against Python 3, then we should be able to use pipenv when running them against Python 2.
Note that this patch allows tests to run against Python 3, but there are plenty of issues preventing them from passing. With this patch in place we can start to add Python 3 support to our packages and have the tests running in CI to ensure we don't regress back to just supporting Python 2.
MozReview-Commit-ID: BuU5gZK83hL
IHG: changed taskcluster/ci/source-test/python.yml
This patch allows executing |mach python-test| against Python 3 by specifying the optional |--three| command line option. When this option is present, pipenv will be used to manage a virtual environment using Python 3 and attempt to run the tests. When it is not present, pipenv will not be used, and everything will work as it did before this patch.
My original plan was to use pipenv regardless of the target version of Python, however I encountered several issues running some of the tests against our Python packages. Once all tests have been patched to run against Python 3, then we should be able to use pipenv when running them against Python 2.
Note that this patch allows tests to run against Python 3, but there are plenty of issues preventing them from passing. With this patch in place we can start to add Python 3 support to our packages and have the tests running in CI to ensure we don't regress back to just supporting Python 2.
MozReview-Commit-ID: BuU5gZK83hL
IHG: changed taskcluster/ci/source-test/python.yml
Mach test now creates a structured logger and passes it down into all
of the various test harnesses. Except |mach python-test| doesn't use
structured logging yet, so the argument is not expected.
For now, just accept **kwargs and ignore the logger. Eventually we'll
want to get python tests to use structured logging, and we can use it
at that point.
MozReview-Commit-ID: 8LwdbgI0vqR
Sometimes, one just wants to run a one-off script with access to all
(or most) the libraries available like mozbuild, etc. but without
the weight of the whole virtualenv, which implies having an objdir
setup, etc.
One of my use cases is to run our preprocessor before the objdir is even
setup, and I'd rather not have one automatically created.
The TestMetadata and TestResolver classes aren't technically part of the build
system. The only connection is that they consume some build system output.
The next patch in this series is going to be merging in a bunch of other test
resolving logic from other parts of the tree. Moving this out first allows us
to keep that extra logic out of mozbuild.
MozReview-Commit-ID: 1eq4SjFVCyW
This switches most tests over to use pytest as the runner instead of unittest (taking
advantage of the fact that pytest can run unittest based tests).
There were a couple tests that had failures when swithing to pytest:
config/tests/unit-expandlibs.py
xpcom/idl-parser/xpidl/runtests.py
For these tests, I added a runwith='unittest' argument so that they still run the
same way as before. Once we fix them to use pytest, the unittest logic in mozunit.py
can be deleted.
MozReview-Commit-ID: Gcsz6z8MeOi
This is a quick and dirty hack to get treeherder to show pytest failures. Long term, we might
want to investigate using something like pytest-mozlog. But the benefit of this approach is
we get to keep pytest's fantastic default log format.
MozReview-Commit-ID: Gcsz6z8MeOi
We set up the temporary directory here so that it persists across multiple test invocations, as each
test is run in its own subprocess.
MozReview-Commit-ID: 7SNip54AqEI
Because test_objects was a generator, using it in the condition always returned True,
even if no tests were found. But extending test_objects to the manifest, converts it
to a list. So this patch simply moves the 'no tests' check a bit later on.
MozReview-Commit-ID: JpETWD1WQWH
This formats the marionette-harness python tests to be a regular |mach python-test| suite. Though
we add subsuite=marionette, this is just for automation purposes. The new preferred way to run the
marionette harness tests locally is:
./mach python-test testing/marionette
They will also run if running the full suite.
The mozbase packages.txt file modifies mozlog to use 'setup.py' instead of 'pth'. The reason for
this is that the marionette-harness tests use the pytest_mozlog pytest plugin for formatting
their results (converts pytest format into something resembling the standard tbpl logging format).
In order for this plugin to get picked up however, mozlog's setup.py file needs to be processed.
MozReview-Commit-ID: Ata99evHxbd
This formats the marionette-harness python tests to be a regular |mach python-test| suite. Though
we add subsuite=marionette, this is just for automation purposes. The new preferred way to run the
marionette harness tests locally is:
./mach python-test testing/marionette
They will also run if running the full suite.
The mozbase packages.txt file modifies mozlog to use 'setup.py' instead of 'pth'. The reason for
this is that the marionette-harness tests use the pytest_mozlog pytest plugin for formatting
their results (converts pytest format into something resembling the standard tbpl logging format).
In order for this plugin to get picked up however, mozlog's setup.py file needs to be processed.
MozReview-Commit-ID: Ata99evHxbd
This adds the ability to use manifestparser subsuites to |mach python-test|.
Subsuites are based on the premise of a "default" set that gets run when no
subsuites are explicitly specified. When a test is labelled with a subsuite,
that test is removed from the default set and will only run if that subsuite
is explicitly specified. This will allow us to chunk python unittests out of
'make check' piecemeal. The default set will run in 'make check', and
individual tasks (e.g mozbase), will specify a subsuite explicitly.
The |mach python-test| implementation is slightly different. By default,
subsuites are not considered if developers do not pass in --subsuite. This
means running |mach python-test| without arguments will still run the full set
of tests, and similarly, passing in test paths will *just work*.
If for some reason a developer needs to actually run the default set, a special
"default" subsuite has been create, so they can use
|mach python-test --subsuite default|. This default subsuite is also what 'make
check' will explicitly invoke.
MozReview-Commit-ID: FaHb4nvuoK9
In bug 1308472 we are seeing 'make -k check' fail intermittently with
the only apparent error message something like:
make: *** [check] Error 245
This debug should make it more clear which test (if any) is responsible
for setting the return code in 'mach python-test', and whether or not
'mach python-test' is actually reaching the end of the function.
MozReview-Commit-ID: 6IQrZQqs8ij
The build system's TestResolver does a pretty good job of getting the right manifestparser
based tests to run, but it isn't perfect. Notably, it ignores the 'disabled' key. We filter
the tests through manifestparser here to make sure the build system didn't miss anything.
For context, this is also what the other test harnesses (e.g mochitest) do as well.
MozReview-Commit-ID: FaHb4nvuoK9
This deprecates PYTHON_UNIT_TESTS and replaces it with PYTHON_UNITTEST_MANIFESTS.
In the build system, this means python unittests will be treated the same as all
other test suites that use manifestparser. New manifests called 'python.ini' have
been created for all test directories containing python unittests.
MozReview-Commit-ID: IBHG7Thif2D
We recently switched make check to call into |mach python-test| rather than invoking python
itself for each test file. But this ended up slowing down the tests as they were no longer
being run in parallel. This patch adds a --jobs flag to python-tests and runs test files in
parallel.
Note: if more than one job is used, output per test will be buffered and printed at the end
to avoid interleaving. This has the unfortunate side effect of making |mach python-test| look
like it is hanging, especially if running a very long file like mozbase's test.py. For this
reason, we still use -j1 by default so output will continue to be streamed. In automation we
will use multiple processes though.
MozReview-Commit-ID: 3u0wOFmyQLI
In the `python-test` mach command and the mozharness script for
the Marionette harness tests, use the vendored-in Pytest
instead of installing from pip.
Add the Marionette harness test requirements file to the
file_patterns in the definition of the marionette-harness taskcluster
job, as changes to the requirements should trigger the job to run.
MozReview-Commit-ID: J5pln2WB4GY
This commit simply moves 'testing/eslint' to 'tools/lint/eslint' and the eslint related
mach command from 'python/mach_commands.py' to 'tools/lint/mach_commands.py'. It shouldn't
have any functional change on running eslint, either through mach or taskcluster.
This is in preparation for bug 1258341, to make the diffs there a little easier to read.
MozReview-Commit-ID: K03sn9lv9Lv
So a few changes here:
- node_modules is downloaded using tooltool so that we dont need to rely on external infrastructure.
- We have a npm-shrinkwrap.json file that version locks all of our node packages.
- eslint, eslint-plugin-mozilla etc. are now all installed locally.
In reality this means that we don't hit the network and we don't force users into installing global packages.
./mach eslint --setup has also been improved. We install packages locally and display the path of the user's eslint binary (useful for configuring editors).
eslint-plugin-mozilla has been moved from testing/eslint-plugin-mozilla to /testing/eslint/eslint-plugin-mozilla.
The node_modules directory for eslint and other plugins is located in testing/eslint/.
MozReview-Commit-ID: 4SFSxzka6BS
The most common issue I'm hearing with eslint is people who have an outdated
node installed. This does a quick check to verify the version is high enough
before linting.
MozReview-Commit-ID: Em0jn18OUYo
Currently mach treats the first argument to eslint as the path and moves it to
the end of the arguments but this breaks usage like "mach eslint -f json browser".
It used to be necessary to change to the directory you wanted to lint but now
the .eslintignore is at the top level we just run from the top level. This means
the path argument doesn't need to be special anymore.
