Hello community,
here is the log from the commit of package python-os-testr for openSUSE:Factory checked in at 2015-12-13 09:39:52
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Comparing /work/SRC/openSUSE:Factory/python-os-testr (Old)
and /work/SRC/openSUSE:Factory/.python-os-testr.new (New)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "python-os-testr"
Changes:
--------
--- /work/SRC/openSUSE:Factory/python-os-testr/python-os-testr.changes 2015-10-30 13:42:50.000000000 +0100
+++ /work/SRC/openSUSE:Factory/.python-os-testr.new/python-os-testr.changes 2015-12-13 09:39:59.000000000 +0100
@@ -1,0 +2,28 @@
+Fri Oct 16 07:16:29 UTC 2015 - dmueller@suse.com
+
+- update to 0.4.2:
+ * Better blacklist - tested with Nova
+ * Add whitelist file support
+ * Fix issues with the blacklist file regex generation
+ * Use test_to_run var in no-discover
+ * Minor refactoring to make os_testr more testable
+ * Switch to using autogenerated ChangeLog in docs
+ * Change ignore-errors to ignore_errors
+ * Handle a skipped test without a reason message
+ * Minimize output when --abbreviate is used
+ * Make use of argparse groups and add some tests
+ * Convert file names to regular expressions
+ * Handle incomplete subunit streams
+ * Set min pbr version in setup_requires
+ * update requirements
+ * Add TODO entry for moving away from subprocess in ostestr
+ * Improved docs for os-testr commands
+ * Dogfood things for unit tests
+ * Disable printing percent change on run time by default
+ * Misc Python 3 compatibility fixes
+ * Catch exception trying to extract test time
+ * Fix ValueError in subunit_trace
+ * Add support for having comments in the exclude file
+ * Add TODO file to os-testr
+
+-------------------------------------------------------------------
Old:
----
os-testr-0.1.0.tar.gz
New:
----
os-testr-0.4.2.tar.gz
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Other differences:
------------------
++++++ python-os-testr.spec ++++++
--- /var/tmp/diff_new_pack.msttwq/_old 2015-12-13 09:40:00.000000000 +0100
+++ /var/tmp/diff_new_pack.msttwq/_new 2015-12-13 09:40:00.000000000 +0100
@@ -17,7 +17,7 @@
Name: python-os-testr
-Version: 0.1.0
+Version: 0.4.2
Release: 0
Summary: A testr wrapper to provide functionality for OpenStack projects
License: Apache-2.0
@@ -29,7 +29,7 @@
Requires: python-Babel >= 1.3
Requires: python-python-subunit >= 0.0.18
Requires: python-testrepository >= 0.0.18
-Requires: python-testtools >= 0.9.36
+Requires: python-testtools >= 1.4.0
BuildRoot: %{_tmppath}/%{name}-%{version}-build
%if 0%{?suse_version} && 0%{?suse_version} <= 1110
%{!?python_sitelib: %global python_sitelib %(python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()")}
++++++ os-testr-0.1.0.tar.gz -> os-testr-0.4.2.tar.gz ++++++
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/.coveragerc new/os-testr-0.4.2/.coveragerc
--- old/os-testr-0.1.0/.coveragerc 2015-04-02 22:15:38.000000000 +0200
+++ new/os-testr-0.4.2/.coveragerc 2015-10-01 21:27:31.000000000 +0200
@@ -4,4 +4,4 @@
omit = os_testr/tests/*,os_testr/openstack/*
[report]
-ignore-errors = True
+ignore_errors = True
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/AUTHORS new/os-testr-0.4.2/AUTHORS
--- old/os-testr-0.1.0/AUTHORS 2015-04-02 22:16:01.000000000 +0200
+++ new/os-testr-0.4.2/AUTHORS 2015-10-01 21:27:57.000000000 +0200
@@ -1 +1,9 @@
+Assaf Muller
+Davanum Srinivas
+James Page
+John Griffith
+Kun Huang
Matthew Treinish
+Monty Taylor
+TerryHowe
+Thomas Bechtold
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/ChangeLog new/os-testr-0.4.2/ChangeLog
--- old/os-testr-0.1.0/ChangeLog 2015-04-02 22:16:01.000000000 +0200
+++ new/os-testr-0.4.2/ChangeLog 2015-10-01 21:27:57.000000000 +0200
@@ -1,6 +1,50 @@
CHANGES
=======
+0.4.2
+-----
+
+* Force utf8 encoding on subunit attachments output
+
+0.4.1
+-----
+
+* Better blacklist - tested with Nova
+
+0.4.0
+-----
+
+* Add whitelist file support
+* Fix issues with the blacklist file regex generation
+* Use test_to_run var in no-discover
+* Minor refactoring to make os_testr more testable
+* Switch to using autogenerated ChangeLog in docs
+* Change ignore-errors to ignore_errors
+* Handle a skipped test without a reason message
+* Minimize output when --abbreviate is used
+* Make use of argparse groups and add some tests
+
+0.3.0
+-----
+
+* Convert file names to regular expressions
+* Handle incomplete subunit streams
+* Set min pbr version in setup_requires
+* update requirements
+* Add TODO entry for moving away from subprocess in ostestr
+* Improved docs for os-testr commands
+
+0.2.0
+-----
+
+* Dogfood things for unit tests
+* Disable printing percent change on run time by default
+* Misc Python 3 compatibility fixes
+* Catch exception trying to extract test time
+* Fix ValueError in subunit_trace
+* Add support for having comments in the exclude file
+* Add TODO file to os-testr
+
0.1.0
-----
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/PKG-INFO new/os-testr-0.4.2/PKG-INFO
--- old/os-testr-0.1.0/PKG-INFO 2015-04-02 22:16:01.000000000 +0200
+++ new/os-testr-0.4.2/PKG-INFO 2015-10-01 21:27:57.000000000 +0200
@@ -1,6 +1,6 @@
Metadata-Version: 1.1
Name: os-testr
-Version: 0.1.0
+Version: 0.4.2
Summary: A testr wrapper to provide functionality for OpenStack projects
Home-page: http://www.openstack.org/
Author: OpenStack
@@ -26,13 +26,6 @@
information about the run
* subunit2html: generates a test results html page from a subunit stream
- Release Notes
- =============
-
- 0.1.0
- -----
- * First release which includes: ostestr, subunit-trace, and subunit2html
-
Platform: UNKNOWN
Classifier: Environment :: OpenStack
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/README.rst new/os-testr-0.4.2/README.rst
--- old/os-testr-0.1.0/README.rst 2015-04-02 22:15:38.000000000 +0200
+++ new/os-testr-0.4.2/README.rst 2015-10-01 21:27:31.000000000 +0200
@@ -17,10 +17,3 @@
* subunit-trace: an output filter for a subunit stream which provides useful
information about the run
* subunit2html: generates a test results html page from a subunit stream
-
-Release Notes
-=============
-
-0.1.0
------
- * First release which includes: ostestr, subunit-trace, and subunit2html
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/TODO.rst new/os-testr-0.4.2/TODO.rst
--- old/os-testr-0.1.0/TODO.rst 1970-01-01 01:00:00.000000000 +0100
+++ new/os-testr-0.4.2/TODO.rst 2015-10-01 21:27:31.000000000 +0200
@@ -0,0 +1,19 @@
+Work Items for os-testr
+=======================
+
+Short Term
+----------
+ * Expose all subunit-trace options through ostestr
+ * Add --html option to ostestr to run testr with subunit2html output
+ * Add unit tests
+ * For ostestr test selection api
+ * Response code validation on more argument permutations
+Long Term
+---------
+ * Lock down test selection CLI
+ * When this is done it will become release 1.0.0
+ * Add subunit-trace functional tests
+ ** Sample subunit streams and test output from subunit-trace
+ * Add testing for subunit2html
+ * Stop using subprocess in ostestr, everything it uses is python so there
+ isn't a need to shell out for everything.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/doc/source/conf.py new/os-testr-0.4.2/doc/source/conf.py
--- old/os-testr-0.1.0/doc/source/conf.py 2015-04-02 22:15:38.000000000 +0200
+++ new/os-testr-0.4.2/doc/source/conf.py 2015-10-01 21:27:31.000000000 +0200
@@ -22,7 +22,7 @@
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = [
'sphinx.ext.autodoc',
- #'sphinx.ext.intersphinx',
+ # 'sphinx.ext.intersphinx',
'oslosphinx'
]
@@ -38,7 +38,7 @@
# General information about the project.
project = u'os-testr'
-copyright = u'2013, OpenStack Foundation'
+copyright = u'2015, Matthew Treinish'
# If true, '()' will be appended to :func: etc. cross-reference text.
add_function_parentheses = True
@@ -68,8 +68,15 @@
('index',
'%s.tex' % project,
u'%s Documentation' % project,
- u'OpenStack Foundation', 'manual'),
+ u'Matthew Treinish', 'manual'),
]
+man_pages = [('ostestr', 'ostestr', 'tooling to run OpenStack tests',
+ ['Matthew Treinish'], 1),
+ ('subunit_trace', 'subunit-trace', 'pretty output filter for '
+ 'subunit streams', ['Matthew Treinish'], 1),
+ ('subunit2html', 'subunit2html', 'generate a html results page '
+ 'from a subunit stream', ['Matthew Treinish'], 1)]
+
# Example configuration for intersphinx: refer to the Python standard library.
-#intersphinx_mapping = {'http://docs.python.org/': None}
+# intersphinx_mapping = {'http://docs.python.org/': None}
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/doc/source/history.rst new/os-testr-0.4.2/doc/source/history.rst
--- old/os-testr-0.1.0/doc/source/history.rst 1970-01-01 01:00:00.000000000 +0100
+++ new/os-testr-0.4.2/doc/source/history.rst 2015-10-01 21:27:31.000000000 +0200
@@ -0,0 +1 @@
+.. include:: ../../ChangeLog
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/doc/source/index.rst new/os-testr-0.4.2/doc/source/index.rst
--- old/os-testr-0.1.0/doc/source/index.rst 2015-04-02 22:15:38.000000000 +0200
+++ new/os-testr-0.4.2/doc/source/index.rst 2015-10-01 21:27:31.000000000 +0200
@@ -15,6 +15,8 @@
installation
usage
contributing
+ todo
+ history
Indices and tables
==================
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/doc/source/ostestr.rst new/os-testr-0.4.2/doc/source/ostestr.rst
--- old/os-testr-0.1.0/doc/source/ostestr.rst 1970-01-01 01:00:00.000000000 +0100
+++ new/os-testr-0.4.2/doc/source/ostestr.rst 2015-10-01 21:27:31.000000000 +0200
@@ -0,0 +1,217 @@
+.. _ostestr:
+
+ostestr
+=======
+
+The ostestr command provides a wrapper around the testr command included in
+the testrepository package. It's designed to build on the functionality
+included in testr and workaround several UI bugs in the short term. By default
+it also has output that is much more useful for OpenStack's test suites which
+are lengthy in both runtime and number of tests. Please note that the CLI
+semantics are still a work in progress as the project is quite young, so
+default behavior might change in future version.
+
+Summary
+-------
+ ostestr [-b|--blacklist_file ] [-r|--regex REGEX]
+ [-p|--pretty] [--no-pretty] [-s|--subunit] [-l|--list]
+ [-n|--no-discover ] [--slowest] [--no-slowest]
+ [--pdb ] [--parallel] [--serial]
+ [-c|--concurrency <workers>] [--until-failure] [--print-exclude]
+
+Options
+-------
+
+ --blacklist_file BLACKLIST_FILE, -b BLACKLIST_FILE
+ Path to a blacklist file, this file contains a
+ separate regex exclude on each newline
+ --regex REGEX, -r REGEX
+ A normal testr selection regex. If a blacklist file is
+ specified, the regex will be appended to the end of
+ the generated regex from that file
+ --pretty, -p
+ Print pretty output from subunit-trace. This is
+ mutually exclusive with --subunit
+ --no-pretty
+ Disable the pretty output with subunit-trace
+ --subunit, -s
+ output the raw subunit v2 from the test run this is
+ mutuall exclusive with --pretty
+ --list, -l
+ List all the tests which will be run.
+ --no-discover TEST_ID, -n TEST_ID
+ Takes in a single test to bypasses test discover and
+ just excute the test specified
+ --slowest
+ After the test run print the slowest tests
+ --no-slowest
+ After the test run don't print the slowest tests
+ --pdb TEST_ID
+ Run a single test that has pdb traces added
+ --parallel
+ Run tests in parallel (this is the default)
+ --serial
+ Run tests serially
+ --concurrency WORKERS, -c WORKERS
+ The number of workers to use when running in parallel.
+ By default this is the number of cpus
+ --until-failure
+ Run the tests in a loop until a failure is
+ encountered. Running with subunit or prettyoutput
+ enable will force the loop to run testsserially
+ --print-exclude
+ If an exclude file is used this option will prints the
+ comment from the same line and all skipped tests
+ before the test run
+
+Running Tests
+-------------
+
+os-testr is primarily for running tests at it's basic level you just invoke
+ostestr to run a test suite for a project. (assuming it's setup to run tests
+using testr already) For example::
+
+ $ ostestr
+
+This will run tests in parallel (with the number of workers matching the number
+of CPUs) and with subunit-trace output. If you need to run tests in serial you
+can use the serial option::
+
+ $ ostestr --serial
+
+Or if you need to adjust the concurrency but still run in parallel you can use
+-c/--concurrency::
+
+ $ ostestr --concurrency 2
+
+If you only want to run an individual test module or more specific (a single
+class, or test) and parallel execution doesn't matter, you can use the
+-n/--no-discover to skip test discovery and just directly calls subunit.run on
+the tests under the covers. Bypassing discovery is desirable when running a
+small subset of tests in a larger test suite because the discovery time can
+often far exceed the total run time of the tests.
+
+For example::
+
+ $ ostestr --no-discover test.test_thing.TestThing.test_thing_method
+
+Additionally, if you need to run a single test module, class, or single test
+with pdb enabled you can use --pdb to directly call testtools.run under the
+covers which works with pdb. For example::
+
+ $ ostestr --pdb tests.test_thing.TestThing.test_thing_method
+
+
+Test Selection
+--------------
+
+ostestr is designed to build on top of the test selection in testr. testr only
+exposed a regex option to select tests. This equivalent is functionality is
+exposed via the --regex option. For example::
+
+ $ ostestr --regex 'magic\.regex'
+
+This will do a straight passthrough of the provided regex to testr.
+Additionally, ostestr allows you to specify a a blacklist file to define a set
+of regexes to exclude. You can specify a blacklist file with the
+--blacklist-file/-b option, for example::
+
+ $ ostestr --blacklist_file $path_to_file
+
+The format for the file is line separated regex, with '#' used to signify the
+start of a comment on a line. For example::
+
+ # Blacklist File
+ ^regex1 # Excludes these tests
+ .*regex2 # exclude those tests
+
+Will generate a regex to pass to testr which will exclude both any tests
+matching '^regex1' and '.*regex2'. If a blacklist file is used in conjunction
+with the --regex option the regex specified with --regex will be appended to
+the generated output from the --blacklist_file. Also it's worth noting that the
+regex test selection options can not be used in conjunction with the
+--no-discover or --pdb options described in the previous section. This is
+because the regex selection requires using testr under the covers to actually
+do the filtering, and those 2 options do not use testr.
+
+It's also worth noting that you can use the test list option to dry run any
+selection arguments you are using. You just need to use --list/-l with your
+selection options to do this, for example::
+
+ $ ostestr --regex 'regex3.*' --blacklist_file blacklist.txt --list
+
+This will list all the tests which will be run by ostestr using that combination
+of arguments.
+
+Please not that all of this selection functionality will be expanded on in the
+future and a default grammar for selecting multiple tests will be chosen in a
+future release. However as of right now all current arguments (which have
+guarantees on always remaining in place) are still required to perform any
+selection logic while this functionality is still under development.
+
+
+Output Options
+--------------
+
+By default ostestr will use subunit-trace as the output filter on the test
+run. It will also print the slowest tests from the run after the run is
+concluded. You can disable the printing the slowest tests with the --no-slowest
+flag, for example::
+
+ $ ostestr --no-slowest
+
+If you'd like to disable the subunit-trace output you can do this using
+--no-pretty::
+
+ $ ostestr --no-pretty
+
+ostestr also provides the option to just output the raw subunit stream on
+STDOUT with --subunit/-s. Note if you want to use this you also have to
+specify --no-pretty as the subunit-trace output and the raw subunit output
+are mutually exclusive. For example, to get raw subunit output the arguments
+would be::
+
+ $ ostestr --no-pretty --subunit
+
+An additional option on top of the blacklist file is --print-exclude option.
+When this option is specified when using a blacklist file before the tests are
+run ostestr will print all the tests it will be excluding from the blacklist
+file. If a line in the blacklist file has a comment that will be printed before
+listing the tests which will be excluded by that line's regex. If no comment is
+present on a line the regex from that line will be used instead. For example,
+if you were using the example blacklist file from the previous section the
+output before the regular test run output would be::
+
+ $ ostestr -b blacklist-file blacklist.txt --print-exclude
+ Excludes these tests
+ regex1_match
+ regex1_exclude
+
+ exclude those tests
+ regex2_match
+ regex2_exclude
+
+ ...
+
+Notes for running with tox
+--------------------------
+
+If you use `tox`_ for running your tests and call ostestr as the test command
+.. _tox: https://tox.readthedocs.org/en/latest/
+it's recommended that you set a posargs following ostestr on the commands
+ stanza. For example::
+
+ [testenv]
+ commands = ostestr {posargs}
+
+this will enable end users to pass args to configure the output, use the
+selection logic, or any other options directly from the tox cli. This will let
+tox take care of the venv management and the environment separation but enable
+direct access to all of the ostestr options to easily customize your test run.
+For example, assuming the above posargs usage you would be to do::
+
+ $ tox -epy34 -- --regex ^regex1
+
+or to skip discovery::
+
+ $ tox -epy34 -- -n test.test_thing.TestThing.test_thing_method
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/doc/source/subunit2html.rst new/os-testr-0.4.2/doc/source/subunit2html.rst
--- old/os-testr-0.1.0/doc/source/subunit2html.rst 1970-01-01 01:00:00.000000000 +0100
+++ new/os-testr-0.4.2/doc/source/subunit2html.rst 2015-10-01 21:27:31.000000000 +0200
@@ -0,0 +1,33 @@
+.. _subunit2html:
+
+subunit2html
+============
+
+subunit2html is a tool that takes in a subunit stream file and will output an
+html page
+
+Summary
+-------
+
+ subunit2html subunit_stream [output]
+
+Usage
+-----
+
+subunit2html takes in 1 mandatory argument. This is used to specify the location
+of the subunit stream file. For example::
+
+ $ subunit2html subunit_stream
+
+By default subunit2html will store the generated html results file at
+results.html file in the current working directory.
+
+An optional second argument can be provided to set the output path of the html
+results file that is generated. If it is provided this will be the output path
+for saving the generated file, otherwise results.html in the current working
+directory will be used. For example::
+
+ $ subunit2html subunit_stream test_results.html
+
+will write the generated html results file to test_results.html in the current
+working directory
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/doc/source/subunit_trace.rst new/os-testr-0.4.2/doc/source/subunit_trace.rst
--- old/os-testr-0.1.0/doc/source/subunit_trace.rst 1970-01-01 01:00:00.000000000 +0100
+++ new/os-testr-0.4.2/doc/source/subunit_trace.rst 2015-10-01 21:27:31.000000000 +0200
@@ -0,0 +1,110 @@
+.. _subunit_trace:
+
+subunit-trace
+=============
+
+subunit-trace is an output filter for subunit streams. It is often used in
+conjunction with test runners that emit subunit to enable a consistent and
+useful realtime output from a test run.
+
+Summary
+-------
+
+subunit-trace [--fails|-f] [--failonly] [--perc-diff|-d] [--no-summary]
+ [--diff-threshold|-t <threshold>]
+
+Options
+-------
+
+ --no-failure-debug, -n
+ Disable printing failure debug information in realtime
+ --fails, -f
+ Print failure debug information after the stream is
+ proccesed
+ --failonly
+ Don't print success items
+ --perc-diff, -d
+ Print percent change in run time on each test
+ --diff-threshold THRESHOLD, -t THRESHOLD
+ Threshold to use for displaying percent change from the
+ avg run time. If one is not specified the percent
+ change will always be displayed.
+ --no-summary
+ Don't print the summary of the test run after completes
+
+Usage
+-----
+subunit-trace will take a subunit stream in via STDIN. This is the only input
+into the tool. It will then print on STDOUT the formatted test result output
+for the test run information contained in the stream.
+
+A subunit v2 stream must be passed into subunit-trace. If only a subunit v1
+stream is available you must use the subunit-1to2 utility to convert it before
+passing the stream into subunit-trace. For example this can be done by chaining
+pipes::
+
+ $ cat subunit_v1 | subunit-1to2 | subunit-trace
+
+Adjusting per test output
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+subunit-trace provides several options to customize it's output. This allows
+users to customize the output from subunit-trace to suit their needs. The output
+from subunit-trace basically comes in 2 parts, the per test output, and the
+summary at the end. By default subunit-trace will print failure messages during
+the per test output, meaning when a test fails it will also print the message
+and any traceback and other attachments at that time. However this can be
+disabled by using --no-failure-debug, -n. For example::
+
+ $ testr run --subunit | subunit-trace --no-failure-debug
+
+Rhere is also the option to print all failures together at the end of the test
+run before the summary view. This is done using the --fails/-f option. For
+example::
+
+ $ testr run --subunit | subunit-trace --fails
+
+Often the --fails and --no-failure-debug options are used in conjunction to
+only print failures at the end of a test run. This is useful for large test
+suites where an error message might be lost in the noise. To do this ::
+
+ $ testr run --subunit | subunit-trace --fails --no-failure-debug
+
+By default subunit-trace will print a line for each test after it completes with
+the test status. However, if you only want to see the run time output for
+failures and not any other test status you can use the --failonly option. For
+example::
+
+ $ testr run --subunit | subunit-trace --failonly
+
+The last output option provided by subunit-trace is to diable the summary view
+of the test run which is normally displayed at the end of a run. You can do
+this using the --no-summary option. For example::
+
+ $ testr run --subunit | subunit-trace --no-summary
+
+
+Show per test run time percent change
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+subunit-trace provides an option to display the percent change in run time
+from the previous run. To do this subunit-trace leverages the testr internals
+a bit. It uses the times.dbm database which, the file repository type in
+testrepository will create, to get the previous run time for a test. If testr
+hasn't ever been used before or for whatever reason subunit-trace is unable to
+find the times.dbm file from testr no percentages will be displayed even if it's
+enabled. Additionally, if a test is run which does not have an entry in the
+times.dbm file will not have a percentage printed for it.
+
+To enable this feature you use --perc-diff/-d, for example::
+
+ $ testr run --subunit | subunit-trace --perc-diff
+
+There is also the option to set a threshold value for this option. If used it
+acts as an absolute value and only percentage changes that exceed it will be
+printed. Use the --diff-threshold/-t option to set a threshold, for example::
+
+ $ testr run --subunit | subunit-trace --perc-diff --threshold 45
+
+This will only display percent differences when the change in run time is either
+>=45% faster or <=45% slower.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/doc/source/todo.rst new/os-testr-0.4.2/doc/source/todo.rst
--- old/os-testr-0.1.0/doc/source/todo.rst 1970-01-01 01:00:00.000000000 +0100
+++ new/os-testr-0.4.2/doc/source/todo.rst 2015-10-01 21:27:31.000000000 +0200
@@ -0,0 +1 @@
+.. include:: ../../TODO.rst
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/doc/source/usage.rst new/os-testr-0.4.2/doc/source/usage.rst
--- old/os-testr-0.1.0/doc/source/usage.rst 2015-04-02 22:15:38.000000000 +0200
+++ new/os-testr-0.4.2/doc/source/usage.rst 2015-10-01 21:27:31.000000000 +0200
@@ -1,7 +1,12 @@
-========
+=====
Usage
-========
+=====
-To use os-testr in a project::
+This section contains the documentation for each of tools packaged in os-testr
- import os_testr
+.. toctree::
+ :maxdepth: 2
+
+ ostestr
+ subunit_trace
+ subunit2html
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/os_testr/os_testr.py new/os-testr-0.4.2/os_testr/os_testr.py
--- old/os-testr-0.1.0/os_testr/os_testr.py 2015-04-02 22:15:38.000000000 +0200
+++ new/os-testr-0.4.2/os_testr/os_testr.py 2015-10-01 21:27:31.000000000 +0200
@@ -23,41 +23,54 @@
from testtools import run as testtools_run
-def parse_args():
+def get_parser(args):
parser = argparse.ArgumentParser(
description='Tool to run openstack tests')
- parser.add_argument('--blacklist_file', '-b',
- help='Path to a blacklist file, this file contains a'
- ' separate regex exclude on each newline')
- parser.add_argument('--regex', '-r',
- help='A normal testr selection regex. If a blacklist '
- 'file is specified, the regex will be appended '
- 'to the end of the generated regex from that '
- 'file')
- parser.add_argument('--pretty', '-p', dest='pretty', action='store_true',
+ list_files = parser.add_mutually_exclusive_group()
+ list_files.add_argument('--blacklist_file', '-b',
+ help='Path to a blacklist file, this file '
+ 'contains a separate regex exclude on each '
+ 'newline')
+ list_files.add_argument('--whitelist_file', '-w',
+ help='Path to a whitelist file, this file '
+ 'contains a separate regex on each newline.')
+ group = parser.add_mutually_exclusive_group()
+ group.add_argument('--regex', '-r',
+ help='A normal testr selection regex. If a blacklist '
+ 'file is specified, the regex will be appended '
+ 'to the end of the generated regex from that '
+ 'file.')
+ group.add_argument('--path', metavar='FILE_OR_DIRECTORY',
+ help='A file name or directory of tests to run.')
+ group.add_argument('--no-discover', '-n', metavar='TEST_ID',
+ help="Takes in a single test to bypasses test "
+ "discover and just excute the test specified. "
+ "A file name may be used in place of a test "
+ "name.")
+ pretty = parser.add_mutually_exclusive_group()
+ pretty.add_argument('--pretty', '-p', dest='pretty', action='store_true',
help='Print pretty output from subunit-trace. This is '
'mutually exclusive with --subunit')
- parser.add_argument('--no-pretty', dest='pretty', action='store_false',
+ pretty.add_argument('--no-pretty', dest='pretty', action='store_false',
help='Disable the pretty output with subunit-trace')
parser.add_argument('--subunit', '-s', action='store_true',
help='output the raw subunit v2 from the test run '
- 'this is mutuall exclusive with --pretty')
+ 'this is mutually exclusive with --pretty')
parser.add_argument('--list', '-l', action='store_true',
help='List all the tests which will be run.')
- parser.add_argument('--no-discover', '-n', metavar='TEST_ID',
- help="Takes in a single test to bypasses test "
- "discover and just excute the test specified")
- parser.add_argument('--slowest', dest='slowest', action='store_true',
- help="after the test run print the slowest tests")
- parser.add_argument('--no-slowest', dest='slowest', action='store_false',
- help="after the test run don't print the slowest "
- "tests")
+ slowest = parser.add_mutually_exclusive_group()
+ slowest.add_argument('--slowest', dest='slowest', action='store_true',
+ help="after the test run print the slowest tests")
+ slowest.add_argument('--no-slowest', dest='slowest', action='store_false',
+ help="after the test run don't print the slowest "
+ "tests")
parser.add_argument('--pdb', metavar='TEST_ID',
help='Run a single test that has pdb traces added')
- parser.add_argument('--parallel', dest='parallel', action='store_true',
- help='Run tests in parallel (this is the default)')
- parser.add_argument('--serial', dest='parallel', action='store_false',
- help='Run tests serially')
+ parallel = parser.add_mutually_exclusive_group()
+ parallel.add_argument('--parallel', dest='parallel', action='store_true',
+ help='Run tests in parallel (this is the default)')
+ parallel.add_argument('--serial', dest='parallel', action='store_false',
+ help='Run tests serially')
parser.add_argument('--concurrency', '-c', type=int, metavar='WORKERS',
help='The number of workers to use when running in '
'parallel. By default this is the number of cpus')
@@ -66,24 +79,89 @@
'encountered. Running with subunit or pretty'
'output enable will force the loop to run tests'
'serially')
+ parser.add_argument('--print-exclude', action='store_true',
+ help='If an exclude file is used this option will '
+ 'prints the comment from the same line and all '
+ 'skipped tests before the test run')
parser.set_defaults(pretty=True, slowest=True, parallel=True)
- opts = parser.parse_args()
- return opts
+ return parser.parse_args(args)
-def construct_regex(blacklist_file, regex):
+def _get_test_list(regex, env=None):
+ env = env or copy.deepcopy(os.environ)
+ proc = subprocess.Popen(['testr', 'list-tests', regex], env=env,
+ stdout=subprocess.PIPE)
+ out = proc.communicate()[0]
+ raw_test_list = out.split('\n')
+ bad = False
+ test_list = []
+ exclude_list = ['OS_', 'CAPTURE', 'TEST_TIMEOUT', 'PYTHON',
+ 'subunit.run discover']
+ for line in raw_test_list:
+ for exclude in exclude_list:
+ if exclude in line:
+ bad = True
+ break
+ elif not line:
+ bad = True
+ break
+ if not bad:
+ test_list.append(line)
+ bad = False
+ return test_list
+
+
+def print_skips(regex, message):
+ test_list = _get_test_list(regex)
+ if test_list:
+ if message:
+ print(message)
+ else:
+ print('Skipped because of regex %s:' % regex)
+ for test in test_list:
+ print(test)
+ # Extra whitespace to separate
+ print('\n')
+
+
+def path_to_regex(path):
+ root, _ = os.path.splitext(path)
+ return root.replace('/', '.')
+
+
+def get_regex_from_whitelist_file(file_path):
+ return '|'.join(open(file_path).read().splitlines())
+
+
+def construct_regex(blacklist_file, whitelist_file, regex, print_exclude):
if not blacklist_file:
exclude_regex = ''
else:
black_file = open(blacklist_file, 'r')
exclude_regex = ''
for line in black_file:
- regex = line.strip()
- exclude_regex = '|'.join([regex, exclude_regex])
+ raw_line = line.strip()
+ split_line = raw_line.split('#')
+ # Before the # is the regex
+ line_regex = split_line[0].strip()
+ if len(split_line) > 1:
+ # After the # is a comment
+ comment = split_line[1].strip()
+ else:
+ comment = ''
+ if line_regex:
+ if print_exclude:
+ print_skips(line_regex, comment)
+ if exclude_regex:
+ exclude_regex = '|'.join([line_regex, exclude_regex])
+ else:
+ exclude_regex = line_regex
if exclude_regex:
- exclude_regex = "'(?!.*" + exclude_regex + ")"
+ exclude_regex = "^((?!" + exclude_regex + ").)*$"
if regex:
exclude_regex += regex
+ if whitelist_file:
+ exclude_regex += '%s' % get_regex_from_whitelist_file(whitelist_file)
return exclude_regex
@@ -106,25 +184,7 @@
# This workaround is necessary because of lp bug 1411804 it's super hacky
# and makes tons of unfounded assumptions, but it works for the most part
if (subunit or pretty) and until_failure:
- proc = subprocess.Popen(['testr', 'list-tests', regex], env=env,
- stdout=subprocess.PIPE)
- out = proc.communicate()[0]
- raw_test_list = out.split('\n')
- bad = False
- test_list = []
- exclude_list = ['CAPTURE', 'TEST_TIMEOUT', 'PYTHON',
- 'subunit.run discover']
- for line in raw_test_list:
- for exclude in exclude_list:
- if exclude in line:
- bad = True
- break
- elif not line:
- bad = True
- break
- if not bad:
- test_list.append(line)
- bad = False
+ test_list = _get_test_list(regex, env)
count = 0
failed = False
if not test_list:
@@ -193,12 +253,25 @@
testtools_run.main([sys.argv[0], test_id], sys.stdout)
-def call_testtools_run(test_id):
- testtools_run.main([sys.argv[0], test_id], sys.stdout)
+def _select_and_call_runner(opts, exclude_regex):
+ ec = 1
+ if not os.path.isdir('.testrepository'):
+ subprocess.call(['testr', 'init'])
+
+ if not opts.no_discover and not opts.pdb:
+ ec = call_testr(exclude_regex, opts.subunit, opts.pretty, opts.list,
+ opts.slowest, opts.parallel, opts.concurrency,
+ opts.until_failure)
+ else:
+ test_to_run = opts.no_discover or opts.pdb
+ if test_to_run.find('/') != -1:
+ test_to_run = path_to_regex(test_to_run)
+ ec = call_subunit_run(test_to_run, opts.pretty, opts.subunit)
+ return ec
def main():
- opts = parse_args()
+ opts = get_parser(sys.argv[1:])
if opts.pretty and opts.subunit:
msg = ('Subunit output and pretty output cannot be specified at the '
'same time')
@@ -217,17 +290,15 @@
msg = "You can not use until_failure mode with pdb or no-discover"
print(msg)
exit(5)
- exclude_regex = construct_regex(opts.blacklist_file, opts.regex)
- if not os.path.isdir('.testrepository'):
- subprocess.call(['testr', 'init'])
- if not opts.no_discover and not opts.pdb:
- exit(call_testr(exclude_regex, opts.subunit, opts.pretty, opts.list,
- opts.slowest, opts.parallel, opts.concurrency,
- opts.until_failure))
- elif opts.pdb:
- exit(call_testtools_run(opts.pdb))
+ if opts.path:
+ regex = path_to_regex(opts.path)
else:
- exit(call_subunit_run(opts.no_discover, opts.pretty, opts.subunit))
+ regex = opts.regex
+ exclude_regex = construct_regex(opts.blacklist_file,
+ opts.whitelist_file,
+ regex,
+ opts.print_exclude)
+ exit(_select_and_call_runner(opts, exclude_regex))
if __name__ == '__main__':
main()
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/os_testr/subunit_trace.py new/os-testr-0.4.2/os_testr/subunit_trace.py
--- old/os-testr-0.1.0/os_testr/subunit_trace.py 2015-04-02 22:15:38.000000000 +0200
+++ new/os-testr-0.4.2/os_testr/subunit_trace.py 2015-10-01 21:27:32.000000000 +0200
@@ -120,6 +120,7 @@
# indent attachment lines 4 spaces to make them visually
# offset
for line in detail.as_text().split('\n'):
+ line = line.encode('utf8')
stream.write(" %s\n" % line)
@@ -131,7 +132,14 @@
test_times = dbm.open(times_db_path)
except Exception:
return False
- avg_runtime = float(test_times.get(str(test_id), False))
+ try:
+ avg_runtime = float(test_times.get(str(test_id), False))
+ except Exception:
+ try:
+ avg_runtime = float(test_times[str(test_id)])
+ except Exception:
+ avg_runtime = False
+
if avg_runtime and avg_runtime > 0:
run_time = float(run_time.rstrip('s'))
perc_diff = ((run_time - avg_runtime) / avg_runtime) * 100
@@ -140,7 +148,7 @@
def show_outcome(stream, test, print_failures=False, failonly=False,
- threshold='0'):
+ enable_diff=False, threshold='0', abbreviate=False):
global RESULTS
status = test['status']
# TODO(sdague): ask lifeless why on this?
@@ -161,29 +169,45 @@
if status == 'fail':
FAILS.append(test)
- stream.write('{%s} %s [%s] ... FAILED\n' % (
- worker, name, duration))
- if not print_failures:
- print_attachments(stream, test, all_channels=True)
+ if abbreviate:
+ stream.write('F')
+ else:
+ stream.write('{%s} %s [%s] ... FAILED\n' % (
+ worker, name, duration))
+ if not print_failures:
+ print_attachments(stream, test, all_channels=True)
elif not failonly:
if status == 'success':
- out_string = '{%s} %s [%s' % (worker, name, duration)
- perc_diff = find_test_run_time_diff(test['id'], duration)
- if perc_diff and abs(perc_diff) >= abs(float(threshold)):
- if perc_diff > 0:
- out_string = out_string + ' +%.2f%%' % perc_diff
- else:
- out_string = out_string + ' %.2f%%' % perc_diff
- stream.write(out_string + '] ... ok\n')
- print_attachments(stream, test)
+ if abbreviate:
+ stream.write('.')
+ else:
+ out_string = '{%s} %s [%s' % (worker, name, duration)
+ perc_diff = find_test_run_time_diff(test['id'], duration)
+ if enable_diff:
+ if perc_diff and abs(perc_diff) >= abs(float(threshold)):
+ if perc_diff > 0:
+ out_string = out_string + ' +%.2f%%' % perc_diff
+ else:
+ out_string = out_string + ' %.2f%%' % perc_diff
+ stream.write(out_string + '] ... ok\n')
+ print_attachments(stream, test)
elif status == 'skip':
- stream.write('{%s} %s ... SKIPPED: %s\n' % (
- worker, name, test['details']['reason'].as_text()))
+ if abbreviate:
+ stream.write('S')
+ else:
+ reason = test['details'].get('reason', '')
+ if reason:
+ reason = ': ' + reason.as_text()
+ stream.write('{%s} %s ... SKIPPED%s\n' % (
+ worker, name, reason))
else:
- stream.write('{%s} %s [%s] ... %s\n' % (
- worker, name, duration, test['status']))
- if not print_failures:
- print_attachments(stream, test, all_channels=True)
+ if abbreviate:
+ stream.write('%s' % test['status'][0])
+ else:
+ stream.write('{%s} %s [%s] ... %s\n' % (
+ worker, name, duration, test['status']))
+ if not print_failures:
+ print_attachments(stream, test, all_channels=True)
stream.flush()
@@ -220,15 +244,24 @@
runtime = 0.0
for k, v in RESULTS.items():
for test in v:
- runtime += float(get_duration(test['timestamps']).strip('s'))
+ test_dur = get_duration(test['timestamps']).strip('s')
+ # NOTE(toabctl): get_duration() can return an empty string
+ # which leads to a ValueError when casting to float
+ if test_dur:
+ runtime += float(test_dur)
return runtime
def worker_stats(worker):
tests = RESULTS[worker]
num_tests = len(tests)
- delta = tests[-1]['timestamps'][1] - tests[0]['timestamps'][0]
- return num_tests, delta
+ stop_time = tests[-1]['timestamps'][1]
+ start_time = tests[0]['timestamps'][0]
+ if not start_time or not stop_time:
+ delta = 'N/A'
+ else:
+ delta = stop_time - start_time
+ return num_tests, str(delta)
def print_summary(stream, elapsed_time):
@@ -254,8 +287,11 @@
"Race in testr accounting.\n" % w)
else:
num, time = worker_stats(w)
- stream.write(" - Worker %s (%s tests) => %ss\n" %
- (w, num, time))
+ out_str = " - Worker %s (%s tests) => %s" % (w, num, time)
+ if time.isdigit():
+ out_str += 's'
+ out_str += '\n'
+ stream.write(out_str)
def parse_args():
@@ -271,6 +307,12 @@
default=(
os.environ.get('TRACE_FAILONLY', False)
is not False))
+ parser.add_argument('--abbreviate', '-a', action='store_true',
+ dest='abbreviate', help='Print one character status'
+ 'for each test')
+ parser.add_argument('--perc-diff', '-d', action='store_true',
+ dest='enable_diff',
+ help="Print percent change in run time on each test ")
parser.add_argument('--diff-threshold', '-t', dest='threshold',
help="Threshold to use for displaying percent change "
"from the avg run time. If one is not specified "
@@ -288,7 +330,9 @@
outcomes = testtools.StreamToDict(
functools.partial(show_outcome, sys.stdout,
print_failures=args.print_failures,
- failonly=args.failonly))
+ failonly=args.failonly,
+ enable_diff=args.enable_diff,
+ abbreviate=args.abbreviate))
summary = testtools.StreamSummary()
result = testtools.CopyStreamResult([outcomes, summary])
result = testtools.StreamResultRouter(result)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/os_testr/tests/test_os_testr.py new/os-testr-0.4.2/os_testr/tests/test_os_testr.py
--- old/os-testr-0.1.0/os_testr/tests/test_os_testr.py 2015-04-02 22:15:38.000000000 +0200
+++ new/os-testr-0.4.2/os_testr/tests/test_os_testr.py 2015-10-01 21:27:31.000000000 +0200
@@ -19,10 +19,238 @@
Tests for `os_testr` module.
"""
+import mock
+import six
+
+from os_testr import os_testr
from os_testr.tests import base
-class TestOs_testr(base.TestCase):
+class TestPathToRegex(base.TestCase):
+
+ def test_file_name(self):
+ result = os_testr.path_to_regex("tests/network/v2/test_net.py")
+ self.assertEqual("tests.network.v2.test_net", result)
+ result = os_testr.path_to_regex("openstack/tests/network/v2")
+ self.assertEqual("openstack.tests.network.v2", result)
+
+
+class TestGetParser(base.TestCase):
+ def test_pretty(self):
+ namespace = os_testr.get_parser(['--pretty'])
+ self.assertEqual(True, namespace.pretty)
+ namespace = os_testr.get_parser(['--no-pretty'])
+ self.assertEqual(False, namespace.pretty)
+ self.assertRaises(SystemExit, os_testr.get_parser,
+ ['--no-pretty', '--pretty'])
+
+ def test_slowest(self):
+ namespace = os_testr.get_parser(['--slowest'])
+ self.assertEqual(True, namespace.slowest)
+ namespace = os_testr.get_parser(['--no-slowest'])
+ self.assertEqual(False, namespace.slowest)
+ self.assertRaises(SystemExit, os_testr.get_parser,
+ ['--no-slowest', '--slowest'])
+
+ def test_parallel(self):
+ namespace = os_testr.get_parser(['--parallel'])
+ self.assertEqual(True, namespace.parallel)
+ namespace = os_testr.get_parser(['--serial'])
+ self.assertEqual(False, namespace.parallel)
+ self.assertRaises(SystemExit, os_testr.get_parser,
+ ['--parallel', '--serial'])
+
+
+class TestCallers(base.TestCase):
+ def test_no_discover(self):
+ namespace = os_testr.get_parser(['-n', 'project.tests.foo'])
+
+ def _fake_exit(arg):
+ self.assertTrue(arg)
+
+ def _fake_run(*args, **kwargs):
+ return 'project.tests.foo' in args
+
+ with mock.patch.object(os_testr, 'exit', side_effect=_fake_exit), \
+ mock.patch.object(os_testr, 'get_parser', return_value=namespace), \
+ mock.patch.object(os_testr,
+ 'call_subunit_run',
+ side_effect=_fake_run):
+ os_testr.main()
+
+ def test_no_discover_path(self):
+ namespace = os_testr.get_parser(['-n', 'project/tests/foo'])
+
+ def _fake_exit(arg):
+ self.assertTrue(arg)
+
+ def _fake_run(*args, **kwargs):
+ return 'project.tests.foo' in args
+
+ with mock.patch.object(os_testr, 'exit', side_effect=_fake_exit), \
+ mock.patch.object(os_testr, 'get_parser', return_value=namespace), \
+ mock.patch.object(os_testr,
+ 'call_subunit_run',
+ side_effect=_fake_run):
+ os_testr.main()
+
+ def test_pdb(self):
+ namespace = os_testr.get_parser(['--pdb', 'project.tests.foo'])
+
+ def _fake_exit(arg):
+ self.assertTrue(arg)
+
+ def _fake_run(*args, **kwargs):
+ return 'project.tests.foo' in args
+
+ with mock.patch.object(os_testr, 'exit', side_effect=_fake_exit), \
+ mock.patch.object(os_testr, 'get_parser', return_value=namespace), \
+ mock.patch.object(os_testr,
+ 'call_subunit_run',
+ side_effect=_fake_run):
+ os_testr.main()
+
+ def test_pdb_path(self):
+ namespace = os_testr.get_parser(['--pdb', 'project/tests/foo'])
+
+ def _fake_exit(arg):
+ self.assertTrue(arg)
+
+ def _fake_run(*args, **kwargs):
+ return 'project.tests.foo' in args
+
+ with mock.patch.object(os_testr, 'exit', side_effect=_fake_exit), \
+ mock.patch.object(os_testr, 'get_parser', return_value=namespace), \
+ mock.patch.object(os_testr,
+ 'call_subunit_run',
+ side_effect=_fake_run):
+ os_testr.main()
+
+
+class TestConstructRegex(base.TestCase):
+ def test_regex_passthrough(self):
+ result = os_testr.construct_regex(None, None, 'fake_regex', False)
+ self.assertEqual(result, 'fake_regex')
+
+ def test_blacklist_regex_with_comments(self):
+ blacklist_file = six.StringIO()
+ for i in range(4):
+ blacklist_file.write('fake_regex_%s # A Comment\n' % i)
+ blacklist_file.seek(0)
+ with mock.patch('six.moves.builtins.open',
+ return_value=blacklist_file):
+ result = os_testr.construct_regex('fake_path', None, None, False)
+ self.assertEqual(
+ result,
+ "^((?!fake_regex_3|fake_regex_2|fake_regex_1|fake_regex_0).)*$")
+
+ def test_blacklist_regex_without_comments(self):
+ blacklist_file = six.StringIO()
+ for i in range(4):
+ blacklist_file.write('fake_regex_%s\n' % i)
+ blacklist_file.seek(0)
+ with mock.patch('six.moves.builtins.open',
+ return_value=blacklist_file):
+ result = os_testr.construct_regex('fake_path', None, None, False)
+ self.assertEqual(
+ result,
+ "^((?!fake_regex_3|fake_regex_2|fake_regex_1|fake_regex_0).)*$")
+
+ def test_blacklist_regex_with_comments_and_regex(self):
+ blacklist_file = six.StringIO()
+ for i in range(4):
+ blacklist_file.write('fake_regex_%s # Comments\n' % i)
+ blacklist_file.seek(0)
+ with mock.patch('six.moves.builtins.open',
+ return_value=blacklist_file):
+ result = os_testr.construct_regex('fake_path', None,
+ 'fake_regex', False)
+
+ expected_regex = ("^((?!fake_regex_3|fake_regex_2|fake_regex_1|"
+ "fake_regex_0).)*$fake_regex")
+ self.assertEqual(result, expected_regex)
+
+ def test_blacklist_regex_without_comments_and_regex(self):
+ blacklist_file = six.StringIO()
+ for i in range(4):
+ blacklist_file.write('fake_regex_%s\n' % i)
+ blacklist_file.seek(0)
+ with mock.patch('six.moves.builtins.open',
+ return_value=blacklist_file):
+ result = os_testr.construct_regex('fake_path', None,
+ 'fake_regex', False)
+
+ expected_regex = ("^((?!fake_regex_3|fake_regex_2|fake_regex_1|"
+ "fake_regex_0).)*$fake_regex")
+ self.assertEqual(result, expected_regex)
+
+ @mock.patch.object(os_testr, 'print_skips')
+ def test_blacklist_regex_with_comment_print_skips(self, print_mock):
+ blacklist_file = six.StringIO()
+ for i in range(4):
+ blacklist_file.write('fake_regex_%s # Comment\n' % i)
+ blacklist_file.seek(0)
+ with mock.patch('six.moves.builtins.open',
+ return_value=blacklist_file):
+ result = os_testr.construct_regex('fake_path', None,
+ None, True)
+
+ expected_regex = ("^((?!fake_regex_3|fake_regex_2|fake_regex_1|"
+ "fake_regex_0).)*$")
+ self.assertEqual(result, expected_regex)
+ calls = print_mock.mock_calls
+ self.assertEqual(len(calls), 4)
+ args = list(map(lambda x: x[1], calls))
+ self.assertIn(('fake_regex_0', 'Comment'), args)
+ self.assertIn(('fake_regex_1', 'Comment'), args)
+ self.assertIn(('fake_regex_2', 'Comment'), args)
+ self.assertIn(('fake_regex_3', 'Comment'), args)
+
+ @mock.patch.object(os_testr, 'print_skips')
+ def test_blacklist_regex_without_comment_print_skips(self, print_mock):
+ blacklist_file = six.StringIO()
+ for i in range(4):
+ blacklist_file.write('fake_regex_%s\n' % i)
+ blacklist_file.seek(0)
+ with mock.patch('six.moves.builtins.open',
+ return_value=blacklist_file):
+ result = os_testr.construct_regex('fake_path', None,
+ None, True)
+
+ expected_regex = ("^((?!fake_regex_3|fake_regex_2|"
+ "fake_regex_1|fake_regex_0).)*$")
+ self.assertEqual(result, expected_regex)
+ calls = print_mock.mock_calls
+ self.assertEqual(len(calls), 4)
+ args = list(map(lambda x: x[1], calls))
+ self.assertIn(('fake_regex_0', ''), args)
+ self.assertIn(('fake_regex_1', ''), args)
+ self.assertIn(('fake_regex_2', ''), args)
+ self.assertIn(('fake_regex_3', ''), args)
+
+
+class TestWhitelistFile(base.TestCase):
+ def test_read_whitelist_file(self):
+ file_contents = """regex_a
+regex_b"""
+ whitelist_file = six.StringIO()
+ whitelist_file.write(file_contents)
+ whitelist_file.seek(0)
+ with mock.patch('six.moves.builtins.open',
+ return_value=whitelist_file):
+ regex = os_testr.get_regex_from_whitelist_file('/path/to/not_used')
+ self.assertEqual('regex_a|regex_b', regex)
+
+ def test_whitelist_regex_without_comments_and_regex(self):
+ file_contents = """regex_a
+regex_b"""
+ whitelist_file = six.StringIO()
+ whitelist_file.write(file_contents)
+ whitelist_file.seek(0)
+ with mock.patch('six.moves.builtins.open',
+ return_value=whitelist_file):
+ result = os_testr.construct_regex(None, 'fake_path',
+ None, False)
- def test_something(self):
- pass
+ expected_regex = 'regex_a|regex_b'
+ self.assertEqual(result, expected_regex)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/os_testr/tests/test_return_codes.py new/os-testr-0.4.2/os_testr/tests/test_return_codes.py
--- old/os-testr-0.1.0/os_testr/tests/test_return_codes.py 2015-04-02 22:15:38.000000000 +0200
+++ new/os-testr-0.4.2/os_testr/tests/test_return_codes.py 2015-10-01 21:27:31.000000000 +0200
@@ -14,13 +14,13 @@
import os
import shutil
-import StringIO
import subprocess
import tempfile
import testtools
from os_testr.tests import base
+from six import StringIO
DEVNULL = open(os.devnull, 'wb')
@@ -47,8 +47,8 @@
shutil.copy('os_testr/tests/files/setup.cfg', self.setup_cfg_file)
shutil.copy('os_testr/tests/files/__init__.py', self.init_file)
- self.stdout = StringIO.StringIO()
- self.stderr = StringIO.StringIO()
+ self.stdout = StringIO()
+ self.stderr = StringIO()
# Change directory, run wrapper and check result
self.addCleanup(os.chdir, os.path.abspath(os.curdir))
os.chdir(self.directory)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/os_testr/tests/test_subunit_trace.py new/os-testr-0.4.2/os_testr/tests/test_subunit_trace.py
--- old/os-testr-0.1.0/os_testr/tests/test_subunit_trace.py 1970-01-01 01:00:00.000000000 +0100
+++ new/os-testr-0.4.2/os_testr/tests/test_subunit_trace.py 2015-10-01 21:27:31.000000000 +0200
@@ -0,0 +1,61 @@
+# Copyright 2015 SUSE Linux GmbH
+# All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+from datetime import datetime as dt
+
+from ddt import data
+from ddt import ddt
+from ddt import unpack
+from mock import patch
+
+from os_testr import subunit_trace
+from os_testr.tests import base
+
+
+@ddt
+class TestSubunitTrace(base.TestCase):
+
+ @data(([dt(2015, 4, 17, 22, 23, 14, 111111),
+ dt(2015, 4, 17, 22, 23, 14, 111111)],
+ "0.000000s"),
+ ([dt(2015, 4, 17, 22, 23, 14, 111111),
+ dt(2015, 4, 17, 22, 23, 15, 111111)],
+ "1.000000s"),
+ ([dt(2015, 4, 17, 22, 23, 14, 111111),
+ None],
+ ""))
+ @unpack
+ def test_get_durating(self, timestamps, expected_result):
+ self.assertEqual(subunit_trace.get_duration(timestamps),
+ expected_result)
+
+ @data(([dt(2015, 4, 17, 22, 23, 14, 111111),
+ dt(2015, 4, 17, 22, 23, 14, 111111)],
+ 0.0),
+ ([dt(2015, 4, 17, 22, 23, 14, 111111),
+ dt(2015, 4, 17, 22, 23, 15, 111111)],
+ 1.0),
+ ([dt(2015, 4, 17, 22, 23, 14, 111111),
+ None],
+ 0.0))
+ @unpack
+ def test_run_time(self, timestamps, expected_result):
+ patched_res = {
+ 0: [
+ {'timestamps': timestamps}
+ ]
+ }
+ with patch.dict(subunit_trace.RESULTS, patched_res, clear=True):
+ self.assertEqual(subunit_trace.run_time(), expected_result)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/os_testr.egg-info/PKG-INFO new/os-testr-0.4.2/os_testr.egg-info/PKG-INFO
--- old/os-testr-0.1.0/os_testr.egg-info/PKG-INFO 2015-04-02 22:16:01.000000000 +0200
+++ new/os-testr-0.4.2/os_testr.egg-info/PKG-INFO 2015-10-01 21:27:57.000000000 +0200
@@ -1,6 +1,6 @@
Metadata-Version: 1.1
Name: os-testr
-Version: 0.1.0
+Version: 0.4.2
Summary: A testr wrapper to provide functionality for OpenStack projects
Home-page: http://www.openstack.org/
Author: OpenStack
@@ -26,13 +26,6 @@
information about the run
* subunit2html: generates a test results html page from a subunit stream
- Release Notes
- =============
-
- 0.1.0
- -----
- * First release which includes: ostestr, subunit-trace, and subunit2html
-
Platform: UNKNOWN
Classifier: Environment :: OpenStack
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/os_testr.egg-info/SOURCES.txt new/os-testr-0.4.2/os_testr.egg-info/SOURCES.txt
--- old/os-testr-0.1.0/os_testr.egg-info/SOURCES.txt 2015-04-02 22:16:01.000000000 +0200
+++ new/os-testr-0.4.2/os_testr.egg-info/SOURCES.txt 2015-10-01 21:27:57.000000000 +0200
@@ -8,6 +8,7 @@
LICENSE
MANIFEST.in
README.rst
+TODO.rst
babel.cfg
openstack-common.conf
requirements.txt
@@ -17,9 +18,14 @@
tox.ini
doc/source/conf.py
doc/source/contributing.rst
+doc/source/history.rst
doc/source/index.rst
doc/source/installation.rst
+doc/source/ostestr.rst
doc/source/readme.rst
+doc/source/subunit2html.rst
+doc/source/subunit_trace.rst
+doc/source/todo.rst
doc/source/usage.rst
os_testr/__init__.py
os_testr/os_testr.py
@@ -37,6 +43,7 @@
os_testr/tests/base.py
os_testr/tests/test_os_testr.py
os_testr/tests/test_return_codes.py
+os_testr/tests/test_subunit_trace.py
os_testr/tests/files/__init__.py
os_testr/tests/files/failing-tests
os_testr/tests/files/passing-tests
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/os_testr.egg-info/pbr.json new/os-testr-0.4.2/os_testr.egg-info/pbr.json
--- old/os-testr-0.1.0/os_testr.egg-info/pbr.json 2015-04-02 22:16:01.000000000 +0200
+++ new/os-testr-0.4.2/os_testr.egg-info/pbr.json 2015-10-01 21:27:57.000000000 +0200
@@ -1 +1 @@
-{"git_version": "78f9371", "is_release": true}
\ No newline at end of file
+{"git_version": "65e1e18", "is_release": true}
\ No newline at end of file
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/os_testr.egg-info/requires.txt new/os-testr-0.4.2/os_testr.egg-info/requires.txt
--- old/os-testr-0.1.0/os_testr.egg-info/requires.txt 2015-04-02 22:16:01.000000000 +0200
+++ new/os-testr-0.4.2/os_testr.egg-info/requires.txt 2015-10-01 21:27:57.000000000 +0200
@@ -1,5 +1,5 @@
-pbr>=0.6,!=0.7,<1.0
+pbr>=1.3,<2.0
Babel>=1.3
testrepository>=0.0.18
python-subunit>=0.0.18
-testtools>=0.9.36,!=1.2.0
+testtools>=1.4.0
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/requirements.txt new/os-testr-0.4.2/requirements.txt
--- old/os-testr-0.1.0/requirements.txt 2015-04-02 22:15:38.000000000 +0200
+++ new/os-testr-0.4.2/requirements.txt 2015-10-01 21:27:31.000000000 +0200
@@ -2,8 +2,8 @@
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
-pbr>=0.6,!=0.7,<1.0
+pbr>=1.3,<2.0
Babel>=1.3
testrepository>=0.0.18
python-subunit>=0.0.18
-testtools>=0.9.36,!=1.2.0
+testtools>=1.4.0
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/setup.cfg new/os-testr-0.4.2/setup.cfg
--- old/os-testr-0.1.0/setup.cfg 2015-04-02 22:16:01.000000000 +0200
+++ new/os-testr-0.4.2/setup.cfg 2015-10-01 21:27:57.000000000 +0200
@@ -53,7 +53,7 @@
output_file = os_testr/locale/os-testr.pot
[egg_info]
-tag_date = 0
tag_svn_revision = 0
tag_build =
+tag_date = 0
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/setup.py new/os-testr-0.4.2/setup.py
--- old/os-testr-0.1.0/setup.py 2015-04-02 22:15:38.000000000 +0200
+++ new/os-testr-0.4.2/setup.py 2015-10-01 21:27:31.000000000 +0200
@@ -26,5 +26,5 @@
pass
setuptools.setup(
- setup_requires=['pbr'],
+ setup_requires=['pbr>=1.3'],
pbr=True)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/test-requirements.txt new/os-testr-0.4.2/test-requirements.txt
--- old/os-testr-0.1.0/test-requirements.txt 2015-04-02 22:15:38.000000000 +0200
+++ new/os-testr-0.4.2/test-requirements.txt 2015-10-01 21:27:31.000000000 +0200
@@ -10,3 +10,5 @@
oslosphinx>=2.2.0 # Apache-2.0
oslotest>=1.2.0 # Apache-2.0
testscenarios>=0.4
+ddt>=0.4.0
+six>=1.9.0
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-testr-0.1.0/tox.ini new/os-testr-0.4.2/tox.ini
--- old/os-testr-0.1.0/tox.ini 2015-04-02 22:15:38.000000000 +0200
+++ new/os-testr-0.4.2/tox.ini 2015-10-01 21:27:31.000000000 +0200
@@ -10,7 +10,7 @@
VIRTUAL_ENV={envdir}
deps = -r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
-commands = python setup.py testr --slowest --testr-args='{posargs}'
+commands = ostestr {posargs}
[testenv:pep8]
commands = flake8