corysanin/tubestation
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Bug 1108399 - Split mach documentation into multiple articles; r=ahal

Browse Source 
The main mach docs page is a bit long. Let's split it into multiple
articles to increase readability going forward.
This commit is contained in:
Gregory Szorc
2014-12-07 11:34:06 -08:00
parent 39a76e34a5
commit bfe6799d76
4 changed files with 291 additions and 275 deletions
Download Patch File Download Diff File Expand all files Collapse all files

135
python/mach/docs/commands.rst Normal file
View File

@@ -0,0 +1,135 @@
.. _mach_commands:
=====================
Implementing Commands
=====================
Mach commands are defined via Python decorators.
All the relevant decorators are defined in the *mach.decorators* module.
The important decorators are as follows:
:py:func:`CommandProvider <mach.decorators.CommandProvider>`
A class decorator that denotes that a class contains mach
commands. The decorator takes no arguments.
:py:func:`Command <mach.decorators.Command>`
A method decorator that denotes that the method should be called when
the specified command is requested. The decorator takes a command name
as its first argument and a number of additional arguments to
configure the behavior of the command.
:py:func:`CommandArgument <mach.decorators.CommandArgument>`
A method decorator that defines an argument to the command. Its
arguments are essentially proxied to ArgumentParser.add_argument()
Classes with the ``@CommandProvider`` decorator **must** have an
``__init__`` method that accepts 1 or 2 arguments. If it accepts 2
arguments, the 2nd argument will be a
:py:class:`mach.base.CommandContext` instance.
Here is a complete example:
.. code-block:: python
from mach.decorators import (
CommandArgument,
CommandProvider,
Command,
)
@CommandProvider
class MyClass(object):
@Command('doit', help='Do ALL OF THE THINGS.')
@CommandArgument('--force', '-f', action='store_true',
help='Force doing it.')
def doit(self, force=False):
# Do stuff here.
When the module is loaded, the decorators tell mach about all handlers.
When mach runs, it takes the assembled metadata from these handlers and
hooks it up to the command line driver. Under the hood, arguments passed
to the decorators are being used to help mach parse command arguments,
formulate arguments to the methods, etc. See the documentation in the
*mach.base* module for more.
The Python modules defining mach commands do not need to live inside the
main mach source tree.
Conditionally Filtering Commands
================================
Sometimes it might only make sense to run a command given a certain
context. For example, running tests only makes sense if the product
they are testing has been built, and said build is available. To make
sure a command is only runnable from within a correct context, you can
define a series of conditions on the *Command* decorator.
A condition is simply a function that takes an instance of the
:py:func:`mach.decorators.CommandProvider` class as an argument, and
returns ``True`` or ``False``. If any of the conditions defined on a
command return ``False``, the command will not be runnable. The
docstring of a condition function is used in error messages, to explain
why the command cannot currently be run.
Here is an example:
.. code-block:: python
from mach.decorators import (
CommandProvider,
Command,
)
def build_available(cls):
"""The build needs to be available."""
return cls.build_path is not None
@CommandProvider
class MyClass(MachCommandBase):
def __init__(self, build_path=None):
self.build_path = build_path
@Command('run_tests', conditions=[build_available])
def run_tests(self):
# Do stuff here.
It is important to make sure that any state needed by the condition is
available to instances of the command provider.
By default all commands without any conditions applied will be runnable,
but it is possible to change this behaviour by setting
``require_conditions`` to ``True``:
.. code-block:: python
m = mach.main.Mach()
m.require_conditions = True
Minimizing Code in Commands
===========================
Mach command modules, classes, and methods work best when they are
minimal dispatchers. The reason is import bloat. Currently, the mach
core needs to import every Python file potentially containing mach
commands for every command invocation. If you have dozens of commands or
commands in modules that import a lot of Python code, these imports
could slow mach down and waste memory.
It is thus recommended that mach modules, classes, and methods do as
little work as possible. Ideally the module should only import from
the :py:module:`mach` package. If you need external modules, you should
import them from within the command method.
To keep code size small, the body of a command method should be limited
to:
1. Obtaining user input (parsing arguments, prompting, etc)
2. Calling into some other Python package
3. Formatting output
Of course, these recommendations can be ignored if you want to risk
slower performance.
In the future, the mach driver may cache the dispatching information or
have it intelligently loaded to facilitate lazy loading.

51
python/mach/docs/driver.rst Normal file
View File

@@ -0,0 +1,51 @@
.. _mach_driver:
=======
Drivers
=======
Entry Points
============
It is possible to use setuptools' entry points to load commands
directly from python packages. A mach entry point is a function which
returns a list of files or directories containing mach command
providers. e.g.:
.. code-block:: python
def list_providers():
providers = []
here = os.path.abspath(os.path.dirname(__file__))
for p in os.listdir(here):
if p.endswith('.py'):
providers.append(os.path.join(here, p))
return providers
See http://pythonhosted.org/setuptools/setuptools.html#dynamic-discovery-of-services-and-plugins
for more information on creating an entry point. To search for entry
point plugins, you can call
:py:meth:`mach.main.Mach.load_commands_from_entry_point`. e.g.:
.. code-block:: python
mach.load_commands_from_entry_point("mach.external.providers")
Adding Global Arguments
=======================
Arguments to mach commands are usually command-specific. However,
mach ships with a handful of global arguments that apply to all
commands.
It is possible to extend the list of global arguments. In your
*mach driver*, simply call
:py:meth:`mach.main.Mach.add_global_argument`. e.g.:
.. code-block:: python
mach = mach.main.Mach(os.getcwd())
# Will allow --example to be specified on every mach command.
mach.add_global_argument('--example', action='store_true',
help='Demonstrate an example global argument.')

280
python/mach/docs/index.rst
View File

@@ -66,279 +66,9 @@ these to eventually be removed and replaced with generic features so
mach is suitable for anybody to use. Until then, mach may not be the mach is suitable for anybody to use. Until then, mach may not be the
best fit for you. best fit for you.
Implementing Commands .. toctree::
--------------------- :maxdepth: 1
Mach commands are defined via Python decorators. commands
driver
All the relevant decorators are defined in the *mach.decorators* module. logging
The important decorators are as follows:
:py:func:`CommandProvider <mach.decorators.CommandProvider>`
A class decorator that denotes that a class contains mach
commands. The decorator takes no arguments.
:py:func:`Command <mach.decorators.Command>`
A method decorator that denotes that the method should be called when
the specified command is requested. The decorator takes a command name
as its first argument and a number of additional arguments to
configure the behavior of the command.
:py:func:`CommandArgument <mach.decorators.CommandArgument>`
A method decorator that defines an argument to the command. Its
arguments are essentially proxied to ArgumentParser.add_argument()
Classes with the ``@CommandProvider`` decorator **must** have an
``__init__`` method that accepts 1 or 2 arguments. If it accepts 2
arguments, the 2nd argument will be a
:py:class:`mach.base.CommandContext` instance.
Here is a complete example:
.. code-block:: python
from mach.decorators import (
CommandArgument,
CommandProvider,
Command,
)
@CommandProvider
class MyClass(object):
@Command('doit', help='Do ALL OF THE THINGS.')
@CommandArgument('--force', '-f', action='store_true',
help='Force doing it.')
def doit(self, force=False):
# Do stuff here.
When the module is loaded, the decorators tell mach about all handlers.
When mach runs, it takes the assembled metadata from these handlers and
hooks it up to the command line driver. Under the hood, arguments passed
to the decorators are being used to help mach parse command arguments,
formulate arguments to the methods, etc. See the documentation in the
*mach.base* module for more.
The Python modules defining mach commands do not need to live inside the
main mach source tree.
Conditionally Filtering Commands
--------------------------------
Sometimes it might only make sense to run a command given a certain
context. For example, running tests only makes sense if the product
they are testing has been built, and said build is available. To make
sure a command is only runnable from within a correct context, you can
define a series of conditions on the *Command* decorator.
A condition is simply a function that takes an instance of the
:py:func:`mach.decorators.CommandProvider` class as an argument, and
returns ``True`` or ``False``. If any of the conditions defined on a
command return ``False``, the command will not be runnable. The
docstring of a condition function is used in error messages, to explain
why the command cannot currently be run.
Here is an example:
.. code-block:: python
from mach.decorators import (
CommandProvider,
Command,
)
def build_available(cls):
"""The build needs to be available."""
return cls.build_path is not None
@CommandProvider
class MyClass(MachCommandBase):
def __init__(self, build_path=None):
self.build_path = build_path
@Command('run_tests', conditions=[build_available])
def run_tests(self):
# Do stuff here.
It is important to make sure that any state needed by the condition is
available to instances of the command provider.
By default all commands without any conditions applied will be runnable,
but it is possible to change this behaviour by setting
``require_conditions`` to ``True``:
.. code-block:: python
m = mach.main.Mach()
m.require_conditions = True
Minimizing Code in Commands
---------------------------
Mach command modules, classes, and methods work best when they are
minimal dispatchers. The reason is import bloat. Currently, the mach
core needs to import every Python file potentially containing mach
commands for every command invocation. If you have dozens of commands or
commands in modules that import a lot of Python code, these imports
could slow mach down and waste memory.
It is thus recommended that mach modules, classes, and methods do as
little work as possible. Ideally the module should only import from
the :py:module:`mach` package. If you need external modules, you should
import them from within the command method.
To keep code size small, the body of a command method should be limited
to:
1. Obtaining user input (parsing arguments, prompting, etc)
2. Calling into some other Python package
3. Formatting output
Of course, these recommendations can be ignored if you want to risk
slower performance.
In the future, the mach driver may cache the dispatching information or
have it intelligently loaded to facilitate lazy loading.
Logging
=======
Mach configures a built-in logging facility so commands can easily log
data.
What sets the logging facility apart from most loggers you've seen is
that it encourages structured logging. Instead of conventional logging
where simple strings are logged, the internal logging mechanism logs all
events with the following pieces of information:
* A string *action*
* A dict of log message fields
* A formatting string
Essentially, instead of assembling a human-readable string at
logging-time, you create an object holding all the pieces of data that
will constitute your logged event. For each unique type of logged event,
you assign an *action* name.
Depending on how logging is configured, your logged event could get
written a couple of different ways.
JSON Logging
------------
Where machines are the intended target of the logging data, a JSON
logger is configured. The JSON logger assembles an array consisting of
the following elements:
* Decimal wall clock time in seconds since UNIX epoch
* String *action* of message
* Object with structured message data
The JSON-serialized array is written to a configured file handle.
Consumers of this logging stream can just perform a readline() then feed
that into a JSON deserializer to reconstruct the original logged
message. They can key off the *action* element to determine how to
process individual events. There is no need to invent a parser.
Convenient, isn't it?
Logging for Humans
------------------
Where humans are the intended consumer of a log message, the structured
log message are converted to more human-friendly form. This is done by
utilizing the *formatting* string provided at log time. The logger
simply calls the *format* method of the formatting string, passing the
dict containing the message's fields.
When *mach* is used in a terminal that supports it, the logging facility
also supports terminal features such as colorization. This is done
automatically in the logging layer - there is no need to control this at
logging time.
In addition, messages intended for humans typically prepends every line
with the time passed since the application started.
Logging HOWTO
-------------
Structured logging piggybacks on top of Python's built-in logging
infrastructure provided by the *logging* package. We accomplish this by
taking advantage of *logging.Logger.log()*'s *extra* argument. To this
argument, we pass a dict with the fields *action* and *params*. These
are the string *action* and dict of message fields, respectively. The
formatting string is passed as the *msg* argument, like normal.
If you were logging to a logger directly, you would do something like:
.. code-block:: python
logger.log(logging.INFO, 'My name is {name}',
extra={'action': 'my_name', 'params': {'name': 'Gregory'}})
The JSON logging would produce something like::
[1339985554.306338, "my_name", {"name": "Gregory"}]
Human logging would produce something like::
0.52 My name is Gregory
Since there is a lot of complexity using logger.log directly, it is
recommended to go through a wrapping layer that hides part of the
complexity for you. The easiest way to do this is by utilizing the
LoggingMixin:
.. code-block:: python
import logging
from mach.mixin.logging import LoggingMixin
class MyClass(LoggingMixin):
def foo(self):
self.log(logging.INFO, 'foo_start', {'bar': True},
'Foo performed. Bar: {bar}')
Entry Points
============
It is possible to use setuptools' entry points to load commands
directly from python packages. A mach entry point is a function which
returns a list of files or directories containing mach command
providers. e.g.:
.. code-block:: python
def list_providers():
providers = []
here = os.path.abspath(os.path.dirname(__file__))
for p in os.listdir(here):
if p.endswith('.py'):
providers.append(os.path.join(here, p))
return providers
See http://pythonhosted.org/setuptools/setuptools.html#dynamic-discovery-of-services-and-plugins
for more information on creating an entry point. To search for entry
point plugins, you can call
:py:meth:`mach.main.Mach.load_commands_from_entry_point`. e.g.:
.. code-block:: python
mach.load_commands_from_entry_point("mach.external.providers")
Adding Global Arguments
=======================
Arguments to mach commands are usually command-specific. However,
mach ships with a handful of global arguments that apply to all
commands.
It is possible to extend the list of global arguments. In your
*mach driver*, simply call
:py:meth:`mach.main.Mach.add_global_argument`. e.g.:
.. code-block:: python
mach = mach.main.Mach(os.getcwd())
# Will allow --example to be specified on every mach command.
mach.add_global_argument('--example', action='store_true',
help='Demonstrate an example global argument.')

100
python/mach/docs/logging.rst Normal file
View File

@@ -0,0 +1,100 @@
.. _mach_logging:
=======
Logging
=======
Mach configures a built-in logging facility so commands can easily log
data.
What sets the logging facility apart from most loggers you've seen is
that it encourages structured logging. Instead of conventional logging
where simple strings are logged, the internal logging mechanism logs all
events with the following pieces of information:
* A string *action*
* A dict of log message fields
* A formatting string
Essentially, instead of assembling a human-readable string at
logging-time, you create an object holding all the pieces of data that
will constitute your logged event. For each unique type of logged event,
you assign an *action* name.
Depending on how logging is configured, your logged event could get
written a couple of different ways.
JSON Logging
============
Where machines are the intended target of the logging data, a JSON
logger is configured. The JSON logger assembles an array consisting of
the following elements:
* Decimal wall clock time in seconds since UNIX epoch
* String *action* of message
* Object with structured message data
The JSON-serialized array is written to a configured file handle.
Consumers of this logging stream can just perform a readline() then feed
that into a JSON deserializer to reconstruct the original logged
message. They can key off the *action* element to determine how to
process individual events. There is no need to invent a parser.
Convenient, isn't it?
Logging for Humans
==================
Where humans are the intended consumer of a log message, the structured
log message are converted to more human-friendly form. This is done by
utilizing the *formatting* string provided at log time. The logger
simply calls the *format* method of the formatting string, passing the
dict containing the message's fields.
When *mach* is used in a terminal that supports it, the logging facility
also supports terminal features such as colorization. This is done
automatically in the logging layer - there is no need to control this at
logging time.
In addition, messages intended for humans typically prepends every line
with the time passed since the application started.
Logging HOWTO
=============
Structured logging piggybacks on top of Python's built-in logging
infrastructure provided by the *logging* package. We accomplish this by
taking advantage of *logging.Logger.log()*'s *extra* argument. To this
argument, we pass a dict with the fields *action* and *params*. These
are the string *action* and dict of message fields, respectively. The
formatting string is passed as the *msg* argument, like normal.
If you were logging to a logger directly, you would do something like:
.. code-block:: python
logger.log(logging.INFO, 'My name is {name}',
extra={'action': 'my_name', 'params': {'name': 'Gregory'}})
The JSON logging would produce something like::
[1339985554.306338, "my_name", {"name": "Gregory"}]
Human logging would produce something like::
0.52 My name is Gregory
Since there is a lot of complexity using logger.log directly, it is
recommended to go through a wrapping layer that hides part of the
complexity for you. The easiest way to do this is by utilizing the
LoggingMixin:
.. code-block:: python
import logging
from mach.mixin.logging import LoggingMixin
class MyClass(LoggingMixin):
def foo(self):
self.log(logging.INFO, 'foo_start', {'bar': True},
'Foo performed. Bar: {bar}')