Licensed under either the Apache License, Version 2.0 or the BSD 3-clause license at the users choice. A copy of both licenses are available in the project source as Apache-2.0 and BSD. You may not use this file except in compliance with one of these two licences.

Unless required by applicable law or agreed to in writing, software distributed under these licenses is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the license you chose for the specific language governing permissions and limitations under that license.

There are two major revisions of the protocol. Version 1 was trivially humanreadable but had significant defects as far as highly parallel testing wasconcerned - it had no room for doing discovery and execution in parallel,required substantial buffering when multiplexing and was fragile - a corruptbyte could cause an entire stream to be misparsed. Version 1.1 addedencapsulation of binary streams which mitigated some of the issues but thecore remained.

Version 2 shares many of the good characteristics of Version 1 - it can beembedded into a regular text stream (e.g. from a build system) and it stillmodels xUnit style test execution. It also fixes many of the issues withVersion 1 - Version 2 can be multiplexed without excessive buffering (intime or space), it has a well defined recovery mechanism for dealing withcorrupted streams (e.g. where two processes write to the same streamconcurrently, or where the stream generator suffers a bug).

More details on both protocol version s can be found in the 'Protocol' sectionof this document.

Subunit comes with command line filters to process a subunit stream andlanguage bindings for python, C, C++ and shell. Bindings are easy to writefor other languages.

A number of useful things can be done easily with subunit: * Test aggregation: Tests run separately can be combined and then reported/displayed together. For instance, tests from different languages can be shown as a seamless whole, and tests running on multiple machines can be aggregated into a single stream through a multiplexer. * Test archiving: A test run may be recorded and replayed later. * Test isolation: Tests that may crash or otherwise interact badly with each other can be run seperately and then aggregated, rather than interfering with each other or requiring an adhoc test->runner reporting protocol. * Grid testing: subunit can act as the necessary serialisation and deserialiation to get test runs on distributed machines to be reported in real time.

Subunit's language bindings act as integration with various test runners like'check', 'cppunit', Python's 'unittest'. Beyond that a small amount of glue(typically a few lines) will allow Subunit to be used in more sophisticatedways.

Python======

Subunit has excellent Python support: most of the filters and tools are writtenin python and there are facilities for using Subunit to increase test isolationseamlessly within a test suite.

The most common way is to run an existing python test suite and have it outputsubunit via the ``subunit.run`` module::

$ python -m subunit.run mypackage.tests.test_suite

For more information on the Python support Subunit offers , please see``pydoc subunit``, or the source in ``python/subunit/``

C=

Subunit has C bindings to emit the protocol. The 'check' C unit testing projecthas included subunit support in their project for some years now. See'c/README' for more details.

C++===

The C library is includable and usable directly from C++. A TestListener forCPPUnit is included in the Subunit distribution. See 'c++/README' for details.

shell=====

There are two sets of shell tools. There are filters, which accept a subunitstream on stdin and output processed data (or a transformed stream) on stdout.

Then there are unittest facilities similar to those for C : shell bindingsconsisting of simple functions to output protocol elements, and a patch foradding subunit output to the 'ShUnit' shell test runner. See 'shell/README' fordetails.

Filter recipes--------------

To ignore some failing tests whose root cause is already known::

subunit-filter --without 'AttributeError.*flavor'

The xUnit test model--------------------

Subunit implements a slightly modified xUnit test model. The stock standardmodel is that there are tests, which have an id(), can be run, and when runstart, emit an outcome (like success or failure) and then finish.

Subunit extends this with the idea of test enumeration (find out about testsa runner has without running them), tags (allow users to describe tests inways the test framework doesn't apply any semantic value to), file attachments(allow arbitrary data to make analysing a failure easy) and timestamps.

The protocol------------

Version 2, or v2 is new and still under development, but is intended tosupercede version 1 in the very near future. Subunit's bundled tools acceptonly version 2 and only emit version 2, but the new filters subunit-1to2 andsubunit-2to1 can be used to interoperate with older third party libraries.

Version 2=========

Version 2 is a binary protocol consisting of independent packets that can beembedded in the output from tools like make - as long as each packet has noother bytes mixed in with it (which 'make -j N>1' has a tendency of doing).Version 2 is currently in draft form, and early adopters should be willingto either discard stored results (if protocol changes are made), or bulkconvert them back to v1 and then to a newer edition of v2.

The protocol synchronises at the start of the stream, after a packet, orafter any 0x0A byte. That is, a subunit v2 packet starts after a newline ordirectly after the end of the prior packet.

Subunit is intended to be transported over a reliable streaming protocol suchas TCP. As such it does not concern itself with out of order delivery ofpackets. However, because of the possibility of corruption due to eitherbugs in the sender, or due to mixed up data from concurrent writes to the samefd when being embedded, subunit strives to recover reasonably gracefully fromdamaged data.

A key design goal for Subunit version 2 is to allow processing and multiplexingwithout forcing buffering for semantic correctness, as buffering tends to hidehung or otherwise misbehaving tests. That said, limited time based bufferingfor network efficiency is a good idea - this is ultimately implementatorchoice. Line buffering is also discouraged for subunit streams, as droppinginto a debugger or other tool may require interactive traffic even if linebuffering would not otherwise be a problem.

In version two there are two conceptual events - a test status event and a fileattachment event. Events may have timestamps, and the path of multiplexers thatan event is routed through is recorded to permit sending actions back to thesource (such as new tests to run or stdin for driving debuggers and otherinteractive input). Test status events are used to enumerate tests, to reporttests and test helpers as they run. Tests may have tags, used to allowtunnelling extra meanings through subunit without requiring parsing ofarbitrary file attachments. Things that are not standalone tests get markedas such by setting the 'Runnable' flag to false. (For instance, individualassertions in TAP are not runnable tests, only the top level TAP test scriptis runnable).

File attachments are used to provide rich detail about the nature of a failure.File attachments can also be used to encapsulate stdout and stderr both duringand outside tests.

Most numbers are stored in network byte order - Most Significant Byte firstencoded using a variation of http://www.dlugosz.com/ZIP2/VLI.html The firstbyte's top 2 high order bits encode the total number of octets in the number.This encoding can encode values from 0 to 2**30-1, enough to encode ananosecond. Numbers that are not variable length encoded are still stored inMSB order.

All variable length elements of the packet are stored with a length prefixnumber allowing them to be skipped over for consumers that don't need tointerpret them.

UTF-8 strings are with no terminating NUL and should not have any embedded NULs(implementations SHOULD validate any such strings that they process and takesome remedial action (such as discarding the packet as corrupt).

Packets are identified by a single byte signature - 0xB3, which is never legalin a UTF-8 stream as the first byte of a character. 0xB3 starts with the firstbit set and the second not, which is the UTF-8 signature for a continuationbyte. 0xB3 was chosen as 0x73 ('s' in ASCII') with the top two bits replaced bythe 1 and 0 for a continuation byte.

If subunit packets are being embedded in a non-UTF-8 text stream, where 0x73 isa legal character, consider either recoding the text to UTF-8, or usingsubunit's 'file' packets to embed the text stream in subunit, rather than theother way around.

Following the signature byte comes a 16-bit flags field, which includes a4-bit version field - if the version is not 0x2 then the packet cannot beread. It is recommended to signal an error at this point (e.g. by emittinga synthetic error packet and returning to the top level loop to look fornew packets, or exiting with an error). If recovery is desired, treat thepacket signature as an opaque byte and scan for a new synchronisation point.NB: Subunit V1 and V2 packets may legitimately included 0xB3 internally,as they are an 8-bit safe container format, so recovery from this situationmay involve an arbitrary number of false positives until an actual packetis encountered : and even then it may still be false, failing after passingthe version check due to coincidence.

After the flags field is a number field giving the length in bytes for theentire packet including the signature and the checksum. This length mustbe less than 4MiB - 4194303 bytes. The encoding can obviously record a largernumber but one of the goals is to avoid requiring large buffers, or causinglarge latency in the packet forward/processing pipeline. Larger fileattachments can be communicated in multiple packets, and the overhead in such a4MiB packet is approximately 0.2%.

The rest of the packet is a series of optional features as specified by the setfeature bits in the flags field. When absent they are entirely absent.

Forwarding and multiplexing of packets can be done without interpreting theremainder of the packet until the routing code and checksum (which are both atthe end of the packet). Additionally, routers can often avoid copying or movingthe bulk of the packet, as long as the routing code size increase doesn't forcethe length encoding to take up a new byte (which will only happen to packetsless than or equal to 16KiB in length) - large packets are very efficient toroute.

Timestamp when present is a 32 bit unsigned integer for secnods, and a variablelength number for nanoseconds, representing UTC time since Unix Epoch inseconds and nanoseconds.

Test id when present is a UTF-8 string. The test id should uniquely identifyrunnable tests such that they can be selected individually. For tests and otheractions which cannot be individually run (such as testfixtures/layers/subtests) uniqueness is not required (though being humanmeaningful is highly recommended).

Tags when present is a length prefixed vector of UTF-8 strings, one per tag.There are no restrictions on tag content (other than the restrictions on UTF-8strings in subunit in general). Tags have no ordering.

When a MIME type is present, it defines the MIME type for the file across allpackets same file (routing code + testid + name uniquely identifies a file,reset when EOF is flagged). If a file never has a MIME type set, it should betreated as application/octet-stream.

File content when present is a UTF-8 string for the name followed by the lengthin bytes of the content, and then the content octets.

If present routing code is a UTF-8 string. The routing code is used todetermine which test backend a test was running on when doing data analysis,and to route stdin to the test process if interaction is required.

Multiplexers SHOULD add a routing code if none is present, and prefix anyexisting routing code with a routing code ('/' separated) if one is alreadypresent. For example, a multiplexer might label each stream it is multiplexingwith a simple ordinal ('0', '1' etc), and given an incoming packet with routecode '3' from stream '0' would adjust the route code when forwarding the packetto be '0/3'.

Following the end of the packet is a CRC-32 checksum of the contents of thepacket including the signature.

This description is being ported to an EBNF style. Currently its only partly inthat style, but should be fairly clear all the same. When in doubt, refer thesource (and ideally help fix up the description!). Generally the protocol isline orientated and consists of either directives and their parameters, orwhen outside a DETAILS region unexpected lines which are not interpreted bythe parser - they should be forwarded unaltered.

Tags given outside a test are applied to all following testsTags given after a test: line and before the result line for the same testapply only to that test, and inherit the current global tags.A '-' before a tag is used to remove tags - e.g. to prevent a global tagapplying to a single test, or to cancel a global tag.

The progress directive is used to provide progress information about a streamso that stream consumer can provide completion estimates, progress bars and soon. Stream generators that know how many tests will be present in the streamshould output "progress: COUNT". Stream filters that add tests should output"progress: +COUNT", and those that remove tests should output"progress: -COUNT". An absolute count should reset the progress indicators inuse - it indicates that two separate streams from different generators havebeen trivially concatenated together, and there is no knowledge of how manymore complete streams are incoming. Smart concatenation could scan each streamfor their count and sum them, or alternatively translate absolute counts intorelative counts inline. It is recommended that outputters avoid absolute countsunless necessary. The push and pop directives are used to provide local regionsfor progress reporting. This fits with hierarchically operating testenvironments - such as those that organise tests into suites - the top-mostrunner can report on the number of suites, and each suite surround its outputwith a (push, pop) pair. Interpreters should interpret a pop as also advancingthe progress of the restored level by one step. Encountering progressdirectives between the start and end of a test pair indicates that a previoustest was interrupted and did not cleanly terminate: it should be implicitlyclosed with an error (the same as when a stream ends with no closing testdirective for the most recently started test).

The time directive acts as a clock event - it sets the time for all futureevents. The value should be a valid ISO8601 time.

The skip, xfail and uxsuccess outcomes are not supported by all testingenvironments. In Python the testttools (https://launchpad.net/testtools)library is used to translate these automatically if an older Python versionthat does not support them is in use. See the testtools documentation for thetranslation policy.

skip is used to indicate a test was discovered but not executed. xfail is usedto indicate a test that errored in some expected fashion (also know as "TODO"tests in some frameworks). uxsuccess is used to indicate and unexpected successwhere a test though to be failing actually passes. It is complementary toxfail.

Hacking on subunit------------------

Releases========

* Update versions in configure.ac and python/subunit/__init__.py.* Update NEWS.* Do a make distcheck, which will update Makefile etc.* Do a PyPI release: PYTHONPATH=../../python python ../../setup.py sdist upload -s* Upload the regular one to LP.* Push a tagged commit.