Welcome to the haas documentation!¶
haas
is a Python test runner that is backward-compatible with
Python’s built-in unittest Test Cases, but is designed to support more
advanced features, such as project-specific plugins.
Introduction¶
haas
is intended to iron out some of the wrinkles in Python’s
unittest
test runner. haas
features an improved test discover
and loading mechanism, allowing the same base command to be used to
discover tests in packages as well as run individual test cases. In the
future, haas
will feature a plugin system allowing the use of
environment configuration plugins (e.g. configure Windows SxS manifests
or COM before running tests) or even plugins that run code between tests
(e.g. report on threads that are not cleaned up by code under test).
Unlike unittest
, haas
is also usually safe to use within a
project’s source tree as it features more robust detection of the
top-level directory of a project.
Contents:
Getting Started¶
Installing haas
¶
haas
can be easily installed using pip
:
$ pip install haas
For development versions, the source is available from the GitHub
repository. To install haas from GitHub, clone the repository and
install using pip
:
$ git clone https://github.com/sjagoe/haas.git
$ cd haas
$ python setup.py sdist
$ pip install dist/haas*.egg
Using haas
¶
To use the basic test discovery feature of haas
, simply invoke it at
the top-level of a project; this should be enough to detect and run all
unit tests:
$ haas
......................................................................
.................................
----------------------------------------------------------------------
Ran 103 tests in 0.116s
OK
haas
supports some of the same options as Python’s unittest
runner, with one major difference: the starting directory (or package
name) for test discovery is simply specified on the command line:
$ haas haas/tests
......................................................................
.................................
----------------------------------------------------------------------
Ran 103 tests in 0.116s
OK
$ haas haas.tests
......................................................................
.................................
----------------------------------------------------------------------
Ran 103 tests in 0.116s
OK
For the currently available options, use the --help
option:
$ haas --help
usage: haas [-h] [-v | -q] [-f] [-b] [-p PATTERN]
[-t TOP_LEVEL_DIRECTORY]
[--log-level {critical,fatal,error,warning,info,debug}]
[start]
positional arguments:
start Directory or dotted package/module name to start
searching for tests
optional arguments:
-h, --help show this help message and exit
-v, --verbose Verbose output
-q, --quiet Quiet output
-f, --failfast Stop on first fail or error
-b, --buffer Buffer stdout and stderr during tests
-p PATTERN, --pattern PATTERN
Pattern to match tests ('test*.py' default)
-t TOP_LEVEL_DIRECTORY, --top-level-directory TOP_LEVEL_DIRECTORY
Top level directory of project (defaults to start
directory)
--log-level {critical,fatal,error,warning,info,debug}
Log level for haas logging
Discovering tests by a test name only¶
haas
is able to discover a subset of the tests when just a test name
(or any sub-section of a dotted module name) is given on the command
line.
For example, to run all test methods called test_method
, the
following would work:
$ haas -v test_method
test_method (haas.tests._test_cases.TestCase) ... ok
test_method (haas.tests.test_loader.TestCaseSubclass) ... ok
test_method (haas.tests.test_discoverer.FilterTestCase) ... ok
----------------------------------------------------------------------
Ran 3 tests in 0.000s
OK
Note that three tests in three different modules have been run. The
string test_method
is matched at any point in the name
<package>.<module>.<class>.<method>
and across all loadable tests in
the project. To restrict this to a single test, we can use the class
name as an additional matching parameter:
$ haas -v FilterTestCase.test_method
test_method (haas.tests.test_discoverer.FilterTestCase) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
API Reference¶
haas package¶
Subpackages¶
haas.plugins package¶
-
class
haas.plugins.base_hook_plugin.
BaseHookPlugin
(name, enabled, enabling_option)[source]¶ Bases:
haas.plugins.i_hook_plugin.IHookPlugin
The base implementation of hook plugins.
-
class
haas.plugins.discoverer.
Discoverer
(loader, **kwargs)[source]¶ Bases:
haas.plugins.i_discoverer_plugin.IDiscovererPlugin
The
Discoverer
is responsible for finding tests that can be loaded by aLoader
.-
classmethod
add_parser_arguments
(parser, option_prefix, dest_prefix)[source]¶ Add options for the plugin to the main argument parser.
Parameters: - parser (argparse.ArgumentParser) – The parser to extend
- option_prefix (str) – The prefix that option strings added by this plugin should use.
- dest_prefix (str) – The prefix that
dest
strings for options added by this plugin should use.
-
discover
(start, top_level_directory=None, pattern=u'test*.py')[source]¶ Do test case discovery.
This is the top-level entry-point for test discovery.
If the
start
argument is a drectory, thenhaas
will discover all tests in the package contained in that directory.If the
start
argument is not a directory, it is assumed to be a package or module name and tests in the package or module are loaded.FIXME: This needs a better description.
Parameters: - start (str) – The directory, package, module, class or test to load.
- top_level_directory (str) – The path to the top-level directoy of the project. This is the parent directory of the project’stop-level Python package.
- pattern (str) – The glob pattern to match the filenames of modules to search for tests.
-
discover_by_directory
(start_directory, top_level_directory=None, pattern=u'test*.py')[source]¶ Run test discovery in a directory.
Parameters: - start_directory (str) – The package directory in which to start test discovery.
- top_level_directory (str) – The path to the top-level directoy of the project. This is the parent directory of the project’stop-level Python package.
- pattern (str) – The glob pattern to match the filenames of modules to search for tests.
-
discover_by_file
(start_filepath, top_level_directory=None)[source]¶ Run test discovery on a single file.
Parameters: - start_filepath (str) – The module file in which to start test discovery.
- top_level_directory (str) – The path to the top-level directoy of the project. This is the parent directory of the project’stop-level Python package.
-
discover_by_module
(module_name, top_level_directory=None, pattern=u'test*.py')[source]¶ Find all tests in a package or module, or load a single test case if a class or test inside a module was specified.
Parameters: - module_name (str) – The dotted package name, module name or TestCase class and test method.
- top_level_directory (str) – The path to the top-level directoy of the project. This is the parent directory of the project’stop-level Python package.
- pattern (str) – The glob pattern to match the filenames of modules to search for tests.
-
discover_filtered_tests
(filter_name, top_level_directory=None, pattern=u'test*.py')[source]¶ Find all tests whose package, module, class or method names match the
filter_name
string.Parameters: - filter_name (str) – A subsection of the full dotted test name. This can be
simply a test method name (e.g.
test_some_method
), the TestCase class name (e.g.TestMyClass
), a module name (e.g.test_module
), a subpackage (e.g.tests
). It may also be a dotted combination of the above (e.g.TestMyClass.test_some_method
). - top_level_directory (str) – The path to the top-level directoy of the project. This is the parent directory of the project’stop-level Python package.
- pattern (str) – The glob pattern to match the filenames of modules to search for tests.
- filter_name (str) – A subsection of the full dotted test name. This can be
simply a test method name (e.g.
-
discover_single_case
(module, case_attributes)[source]¶ Find and load a single TestCase or TestCase method from a module.
Parameters: - module (module) – The imported Python module containing the TestCase to be loaded.
- case_attributes (list) – A list (length 1 or 2) of str. The first component must be the name of a TestCase subclass. The second component must be the name of a method in the TestCase.
-
classmethod
from_args
(args, arg_prefix, loader)[source]¶ Construct the discoverer from parsed command line arguments.
Parameters: - args (argparse.Namespace) – The
argparse.Namespace
containing parsed arguments. - arg_prefix (str) – The prefix used for arguments beloning solely to this plugin.
- loader (haas.loader.Loader) – The test loader used to construct TestCase and TestSuite instances.
- args (argparse.Namespace) – The
-
classmethod
-
haas.plugins.discoverer.
filter_test_suite
(suite, filter_name)[source]¶ Filter test cases in a test suite by a substring in the full dotted test name.
Parameters: - suite (haas.suite.TestSuite) – The test suite containing tests to be filtered.
- filter_name (str) – The substring of the full dotted name on which to filter. This should not contain a leading or trailing dot.
-
class
haas.plugins.i_result_handler_plugin.
IResultHandlerPlugin
[source]¶ Bases:
object
-
classmethod
add_parser_arguments
(parser, name, option_prefix, dest_prefix)[source]¶ Add options for the plugin to the main argument parser.
Parameters: - parser (argparse.ArgumentParser) – The parser to extend
- name (str) – The name of the plugin.
- option_prefix (str) – The prefix that option strings added by this plugin should use.
- dest_prefix (str) – The prefix that
dest
strings for options added by this plugin should use.
-
classmethod
from_args
(args, name, dest_prefix, test_count)[source]¶ Construct the result handler from parsed command line arguments.
Parameters: - args (argparse.Namespace) – The
argparse.Namespace
containing parsed arguments. - name (str) – The name of the plugin.
- dest_prefix (str) – The prefix that
dest
strings for options added by this plugin should use. - test_count (int) – The totel number of tests discovered.
- args (argparse.Namespace) – The
-
classmethod
-
class
haas.plugins.i_runner_plugin.
IRunnerPlugin
[source]¶ Bases:
object
-
classmethod
add_parser_arguments
(parser, option_prefix, dest_prefix)[source]¶ Add options for the plugin to the main argument parser.
Parameters: - parser (argparse.ArgumentParser) – The parser to extend
- option_prefix (str) – The prefix that option strings added by this plugin should use.
- dest_prefix (str) – The prefix that
dest
strings for options added by this plugin should use.
-
classmethod
-
class
haas.plugins.result_handler.
QuietTestResultHandler
(test_count)[source]¶ Bases:
haas.plugins.i_result_handler_plugin.IResultHandlerPlugin
-
print_error_list
(error_kind, errors)[source]¶ Print the list of errors or failures.
Parameters: - error_kind (str) –
'ERROR'
or'FAIL'
- errors (list) – List of
TestResult
- error_kind (str) –
-
separator1
= u'======================================================================'¶
-
separator2
= u'----------------------------------------------------------------------'¶
-
-
class
haas.plugins.result_handler.
TimingResultHandler
(number_to_summarize)[source]¶ Bases:
haas.plugins.i_result_handler_plugin.IResultHandlerPlugin
-
OPTION_DEFAULT
= <object object>¶
-
separator1
= u'======================================================================'¶
-
separator2
= u'----------------------------------------------------------------------'¶
-
-
class
haas.plugins.result_handler.
VerboseTestResultHandler
(test_count)[source]¶ Bases:
haas.plugins.result_handler.StandardTestResultHandler
-
class
haas.plugins.runner.
BaseTestRunner
(warnings=None)[source]¶ Bases:
haas.plugins.i_runner_plugin.IRunnerPlugin
A test runner class that does not print any output itself.
haas.tests package¶
-
class
haas.tests.builder.
Class
(name, contents=(), bases=(<class 'unittest.case.TestCase'>, ))[source]¶
-
class
haas.tests.builder.
Package
(name, contents=())[source]¶ Bases:
haas.tests.builder.Directory
Submodules¶
haas.error_holder module¶
haas.exceptions module¶
haas.haas_application module¶
-
class
haas.haas_application.
HaasApplication
(argv, **kwargs)[source]¶ Bases:
object
Main haas application entry-point.
-
run
(plugin_manager=None)[source]¶ Run the haas test runner.
This will load and configure the selected plugins, set up the environment and begin test discovery, loading and running.
Parameters: plugin_manager (haas.plugin_manager.PluginManager) – [Optional] Override the use of the default plugin manager.
-
haas.loader module¶
-
class
haas.loader.
Loader
(test_suite_class=None, test_case_class=None, test_method_prefix=u'test', **kwargs)[source]¶ Bases:
object
Load individual test cases from modules and wrap them in the
Suite
container.-
create_suite
(tests=())[source]¶ Create a test suite using the confugured test suite class.
Parameters: tests (sequence) – Sequence of TestCase instances.
-
find_test_method_names
(testcase)[source]¶ Return a list of test method names in the provided
TestCase
subclass.Parameters: testcase (type) – Subclass of unittest.TestCase
-
get_test_cases_from_module
(module)[source]¶ Return a list of TestCase subclasses contained in the provided module object.
Parameters: module (module) – A module object containing TestCases
-
load_case
(testcase)[source]¶ Load a TestSuite containing all TestCase instances for all tests in a TestCase subclass.
Parameters: testcase (type) – A subclass of unittest.TestCase
-
haas.main module¶
haas.module_import_error module¶
haas.plugin_context module¶
haas.plugin_manager module¶
-
class
haas.plugin_manager.
PluginManager
[source]¶ Bases:
object
-
ENVIRONMENT_HOOK
= u'haas.hooks.environment'¶
-
RESULT_HANDLERS
= u'haas.result.handler'¶
-
TEST_DISCOVERY
= u'haas.discovery'¶
-
TEST_RUNNER
= u'haas.runner'¶
-
add_plugin_arguments
(parser)[source]¶ Add plugin arguments to argument parser.
Parameters: parser (argparse.ArgumentParser) – The main haas ArgumentParser.
-
get_driver
(namespace, parsed_args, **kwargs)[source]¶ Get mutually-exlusive plugin for plugin namespace.
-
haas.result module¶
-
class
haas.result.
ResultCollecter
(*args, **kwargs)[source]¶ Bases:
haas.result.ResultCollector
-
class
haas.result.
ResultCollector
(buffer=False, failfast=False)[source]¶ Bases:
object
Collecter for test results. This handles creating
TestResult
instances and handing them off the registered result output handlers.-
addError
(*args, **kw)[source]¶ Register that a test ended in an error.
Parameters: - test (unittest.TestCase) – The test that has completed.
- exception (tuple) –
exc_info
tuple(type, value, traceback)
.
-
addExpectedFailure
(test, exception)[source]¶ Register that a test that failed and was expected to fail.
Parameters: - test (unittest.TestCase) – The test that has completed.
- exception (tuple) –
exc_info
tuple(type, value, traceback)
.
-
addFailure
(*args, **kw)[source]¶ Register that a test ended with a failure.
Parameters: - test (unittest.TestCase) – The test that has completed.
- exception (tuple) –
exc_info
tuple(type, value, traceback)
.
-
addSkip
(test, reason)[source]¶ Register that a test that was skipped.
Parameters: - test (unittest.TestCase) – The test that has completed.
- reason (str) – The reason the test was skipped.
-
addSuccess
(test)[source]¶ Register that a test ended in success.
Parameters: test (unittest.TestCase) – The test that has completed.
-
addUnexpectedSuccess
(*args, **kw)[source]¶ Register a test that passed unexpectedly.
Parameters: test (unittest.TestCase) – The test that has completed.
-
add_result
(result)[source]¶ Add an already-constructed
TestResult
to thisResultCollector
.This may be used when collecting results created by other ResultCollectors (e.g. in subprocesses).
-
separator2
= u'----------------------------------------------------------------------'¶
-
startTest
(test, start_time=None)[source]¶ Indicate that an individual test is starting.
Parameters: - test (unittest.TestCase) – The test that is starting.
- start_time (datetime) – An internal parameter to allow the parallel test runner to set the actual start time of a test run in a subprocess.
-
stop
()[source]¶ Set the
shouldStop
flag, used by the test cases to determine if they should terminate early.
-
-
class
haas.result.
TestCompletionStatus
[source]¶ Bases:
enum.Enum
Enumeration to represent the status of a single test.
-
error
= 3¶ The test encountered an unexpected error.
-
expected_failure
= 5¶ A test failed as expected
-
failure
= 2¶ The test failed, but did not encounter an unexpected error.
-
skipped
= 6¶ A test was skipped
-
success
= 1¶ The test completed successfully.
-
unexpected_success
= 4¶ A test marked as expected to fail unexpected passed.
-
-
class
haas.result.
TestDuration
(start_time, stop_time=None)[source]¶ Bases:
object
An orderable representation of the duration of an individual test.
-
duration
¶
-
start_time
¶
-
stop_time
¶
-
total_seconds
¶
-
-
class
haas.result.
TestResult
(test_class, test_method_name, status, duration, exception=None, message=None)[source]¶ Bases:
object
Container object for all information related to the run of a single test. This contains the test itself, the actual status including the reason or error associated with status, along with timing information.
-
classmethod
from_test_case
(test_case, status, duration, exception=None, message=None, stdout=None, stderr=None)[source]¶ Construct a
TestResult
object from the test and a status.Parameters: - test_case (unittest.TestCase) – The test that this result will represent.
- status (haas.result.TestCompletionStatus) – The status of the test.
- exception (tuple) –
exc_info
tuple(type, value, traceback)
. - message (str) – Optional message associated with the result (e.g. skip reason).
- stdout (str) – The test stdout if stdout was buffered.
- stderr (str) – The test stderr if stderr was buffered.
-
test
¶ The test case instance this result represents.
-
classmethod
haas.suite module¶
-
class
haas.suite.
TestSuite
(tests=())[source]¶ Bases:
object
A
TestSuite
is a container of test cases and allows executing many test cases while managing the state of the overall suite.
-
haas.suite.
find_test_cases
(suite)[source]¶ Generate a list of all test cases contained in a test suite.
Parameters: suite (haas.suite.TestSuite) – The test suite from which to generate the test case list.