Added `./mach python-safety`, distinct from python-test so it doesn't have
to be run on every CI job - its errors may not depend on the area the push has changed.
Added the python/safety directory to ensure a different Pipfile is used, avoiding
conflicts with python-test.
Differential Revision: https://phabricator.services.mozilla.com/D1825
This is a new issue that gets linted with flake8 3.5.0. Basically you should
never use a blank except: statement.
This will catch all exceptions, including KeyboardInterrupt and SystemExit
(which is likely not intended). If a catch all is needed, use
`except: Exception`. If you *really* mean to also catch KeyboardInterrupt et
al, use `except: BaseException`.
Of course, being specific is often better than a catch all.
MozReview-Commit-ID: FKx80MLO4RN
comm-central taskcluster builds have comm-central checkout in `comm/`. This
will add packages specified in comm-central when it checked out, and will
ignore the missing file for mozilla-central builds.
MozReview-Commit-ID: Amg5vJHoWkS
``print()`` has no business being in library code like this. It was
a holdover from this code being copied from bootstrap. So remove it.
While we're here, replace the generic exception with a specific one.
We don't want to be swallowing bugs via ``except Exception``.
MozReview-Commit-ID: 49goUstfPBz
Sparse checkouts may not have all mach_commands.py files.
mach raises an error when a mach_commands.py file is missing.
So, we teach the mach driver to ignore missing file errors when
a sparse checkout is present.
The added code is optimized to avoid an import of mozversioncontrol
and some I/O as part of resolving the repo and VCS binaries because
this file is in the critical path of all mach commands and avoiding
I/O is worthwhile. Since we aren't using sparse checkouts in the
common case, this effectively makes the new code 0 cost.
MozReview-Commit-ID: C6itJga31t5
"mach" in this scope is both the mach module and the driver instance.
Let's change the latter's variable name so we can access the mach
module.
MozReview-Commit-ID: Db6sxDFl2oo
The |mach try| implementation isn't really testing specific so figured while doing this refactor
anyway we could take the opportunity to move it to /tools. This renames the "autotry" module to
"syntax" as one of the 'selectors' under the newly created tryselect module.
MozReview-Commit-ID: DmnGU5sKuCk
Most entries in virtualenv_packages.txt that are .pth or packages.txt
are currently in SEARCH_PATHS in mach_bootstrap. The ones that are
missing would make sense in SEARCH_PATHS. None of non-.pth or
packages.txt entries, however, are in SEARCH_PATHS and don't make sense
there.
On the other hand, virtualenv_packages.txt misses a lot of things that
are in SEARCH_PATHS, all of which should be there. One exception:
xpcom/idl-parser, which causes problems due to the xpidl package
containing an xpidl module, which causes problems with the in-tree
scripts using it. Plus, it needs a cache directory, which is messy, so
it's preferable to keep it away from the virtualenv. It turns out it was
added to mach_bootstrap.py in bug 893976 for a command that was since
then removed (bug 1244736), so we can get away with removing it.
So instead of keeping those two separate lists out of sync, we replace
the SEARCH_PATHS list from mach_bootstrap with one that is derived at
runtime from the contents of virtualenv_packages.txt.
And since a .pth can't fail to install in the virtualenv, it makes no
sense to have psutil.pth defined as optional, which allows it to end up
in the mach_bootstrap search path automatically.
Finally, because we do have overlapping module names in the tree (e.g.
runtests), and mach_bootstrap's SEARCH_PATHS had a guaranteed order,
we change the order of the virtualenv_packages.txt file to match what
used to be in mach_bootstrap, and make all the pth entries use the same
file name so that the order is more guaranteed in the virtualenv too.
usage: mach [global arguments] awsy-test [command arguments]
mach awsy-test runs the in-tree version of the Are We Slim Yet
(AWSY) tests.
awsy-test is implemented as a marionette test and marionette
test arguments also apply although they are not necessary
since reasonable defaults will be chosen.
The AWSY specific arguments can be found in the Command
Arguments for AWSY section below.
awsy-test will automatically download the tp5n.zip talos
pageset from tooltool and install it under
topobjdir/_tests/awsy/html. You can specify your own page set
by specifying --web-root and --page-manifest.
The results of the test will be placed in the results
directory specified by the --results argument.
Command Arguments for AWSY:
--web-root WEBROOTDIR
Path to web server root directory. If not specified,
defaults to topobjdir/_tests/awsy/html.
--page-manifest PAGEMANIFEST
Path to page manifest text file containing a list of
urls to test. The urls must be served from localhost.
If not specified, defaults to
page_load_test/tp5b/tp5n.manifest under the web root.
--results RESULTSDIR Path to results directory. If not specified, defaults
to the parent directory of the web root.
--quick Set --entities=3, --iterations=1, --per-tab-pause=1,
--settle-wait-time=1 for a quick test. Overrides any
explicit argument settings.
--entities ENTITIES Number of urls to load. Defaults to the total number
of urls.
--max-tabs MAXTABS Maximum number of tabs to open. Defaults to 30.
--iterations ITERATIONS
Number of times to run through the test suite.
Defaults to 5.
--per-tab-pause PERTABPAUSE
Seconds to wait in between opening tabs. Defaults to
10.
--settle-wait-time SETTLEWAITTIME
Seconds to wait for things to settled down. Defaults
to 30.
This imports two modules from mozregression in the tree to do so. They
are imported from current trunk on github, rather than the version we
were getting from pypi.
Note we take six from testing/web-platform/tests/tools/six) instead of
moving it to python/six because it's there by coming from a copy of
https://github.com/w3c/wpt-tools, which contains it as a submodule, and
moving it would make updates there harder.
This generalizes the monkey-patching of the main module added
in bug 914563 to allow multiprocessing to be used from config.status
(which triggers the same bug as when it's used with `mach` as main).
MozReview-Commit-ID: AdOdpKzmbsp
Provide a single mach command to automatically generate the static
database of CSS properties that devtools uses for the inspector
and various editors.
MozReview-Commit-ID: 8E2jwxF0KbM
Vendor in Pytest (2.9.2) and its requirement Py (1.4.31),
so that it can be used for e.g. the Marionette
harness unit tests and a pytest plugin for mozlog.
Copy pytest and py package directories (extracted from
tars on Pip) into `mozilla-central/python/`, removing
some support files (e.g. changelog, docs, tests).
Add both `.pth` entries to `virtualenv_packages.txt`.
Add both paths to `SEARCH_PATHS` in `mach_bootstrap.py`.
MozReview-Commit-ID: IOTCOUxX8R9
It used to be there, but was removed in bug 1151124 because what I
thought was the only use of pymake in mach was removed in that bug.
Which, it turns out, was not the case. So restore it, so that mach
empty-makefiles works again.
The wizard has been ported to the version-control-tools repository
and in-tree consumers have been switched to consume it from there. This
code is now dead. Kill it.
References to the now-defunct code have been removed/updated.
MozReview-Commit-ID: 5fpCXdNIp8L
This begins the consolidation of `mach mercurial-setup` into
`mach bootstrap`. The first step is to move the content of the
mach_commands.py file into the bootstrapper's.
I'm not crazy about adding the sys.path entry for tools/mercurial.
I intend to clean this up later.
MozReview-Commit-ID: Cq56wPG8sO1
I never really liked this. Other people had even more visceral
reactions. Let's get rid of it.
The code for touching a file when it runs has also been removed because
the only thing it was used for was the nagging feature.
MozReview-Commit-ID: ERUVkEYgkzx
The `taskgraph` package generates TaskCluster task graphs based on collections
of task "kinds". Initially, there is only one kind, the "legacy" kind, which
reads the YAML files from `testing/taskcluster/tasks` to generate the task
graph.
Try syntax is implemented by filtering the tasks in the taskgraph after it has
been created, then extending the result to include any prerequisite tasks.
A collection of `mach taskgraph` subcommands are provided for developers to
extend or debug the task-graph generation process.
MozReview-Commit-ID: 1TJCns4XxZ8
Runtime configs have been implemented for awhile, but disabled. This patch
enables configuration. Config files will be loaded in the following order
(later files override earlier ones):
1a. $MACHRC
1b. $MOZBUILD_STATE_PATH/machrc (if $MACHRC is unset)
2. topsrcdir/machrc
3. CLI via --settings
Note: .machrc may be used instead of machrc if desired.
MozReview-Commit-ID: IntONAZLGML
This spins up a separate process to submit telemetry data rather than
blocking the execution the current mach command.
Although the initial Python process needs to wait for the second process
to complete prior to exiting, it releases control of the console once it
finishes executing Python code, so from the user's perspective, mahc command
completion is not blocked by submitting telemetry data.
MozReview-Commit-ID: FlKDYd6rNPc
This change moves testing/marionette/client to testing/marionette/harness,
and testing/marionette/driver to testing/marionette/client. This parent
directory name change reflects the true purpose of these Python packages.
MozReview-Commit-ID: B0KrGzDTLlQ
