Test Manager plugin for Trac

Description

Differently from other test management plugins for Trac that use Tickets as test case holders, this one uses Wiki pages and an additional proprietary data model to store Test Cases.
This allows you to not pollute your ticket lists with something that is not a ticket, and at the same time is powered by the Trac search engine and formatting syntax for Wiki pages.

A set of plugins intercept requests for Wiki pages that are test cases and decorate the page with title, breadcrumbs, tree view, type-ahead search inside the catalogs, test case status semaphore and icons and buttons that allow you to create new test cases, sub-catalogs, copy and paste test cases around different catalogs and change a test case status.

Multiple Test Plans can be associated to any Test Catalog, in order to keep track of the execution of the corresponding Test Cases in a particular testing context.

A Programmatic API is available to interact with the test management environment both by an HTTPRESTful interface and a Python classes interface.

The plugin has been made modular:

A generic, customizable Workflow Engine is provided as a separate plugin in the package, which allows you to associate custom workflows to any Resource in Trac, including Wiki pages and any custom object handled through a Resource Manager. This can be done very easily by adding as little as a few lines of code to the component which manages the User interface.

All the data manipulation layer has been extracted into a Generic Persistent Class framework plugin, which can be easily used by any other plugin developer to provide persistence, security, change history, custom properties, pattern matching search, and other features to their basic data model with a handful lines of code.

All of the test objects, i.e. catalogs, test cases, test plans and test cases in a plan (i.e. with a status and a status change history), support:

​Custom properties, which can be declared in the trac.ini file and will be available to the User for change, stored in the database and available to change listeners.

Change history

Listener interface to be notified of object creation, modification and deletion

​Customizable Workflow state machine, declared in the trac.ini file, with the same syntax as for Ticket workflows (I may have reused some existing code here :-)

​Customizable Workflow Operations, via a plugin api so that any component can provide its custom operations to be performed upon any workflow action, as defined in the trac.ini file.

Workflow also supports a listener API for components interested in state transitions and actions performed

Workflow states also support custom properties, so to be able to convey additional context information on a workflow state and use it in listeners or directly from the database.

The developed workflow engine is able to work on any Trac Resource, it is not confined to this plugin ones. You can then define a workflow on any Trac resource, including Wiki pages, declaratively in the trac.ini file.
You will then add a handful of custom code (for example in an ITemplateStreamFilter) to add the markup that the workflow engine generates for you to your desired Trac web page.
See this page for further details.

Here follows an overview of the plugin functionalities. For a brief tutorial, refer to the powerpoint presentations attached below.
Note, anyway that the tutorial was developed for release 1.0.0 and so does not cover Test Plans, Workflows and Custom properties.

Test Catalogs

Test catalogs contain sub-catalogs or Test Cases. A Javascript tree view displays a catalog node and its sub-tree, including all of the test cases contained.

Next to each catalog (or sub-catalog) a number in brackets shows the number of test cases it contains.

Notice at the top of the page breadcrumbs to easily navigate up in the catalogs tree.

You can choose between a tree-like and a tabular representation of the catalog and all of its contained sub-catalogs and test cases, as shown in
The former is useful to actually work on the Test Catalog, adding sub-catalogs and new test cases, or modifying them.
The latter, which also reports a printout of all the Test Case descriptions, may be useful for a Test Engineer to save or print as a handout to actually run the test cases in the catalog.

You can add sub-catalogs or Test Cases simply by entering a name (blanks and case are supported) and click the appropriate button. A new wiki page is generated, with a naming convention allowing the plugin code to position it correctly in the catalogs tree, and opened for browsing.

If you wish to edit the title or add some textual contents to the (sub-)catalog, click on "Edit Page" at the bottom of the screen.

Be careful that the first line (the one surrounded by '==') will always be taken as the title of the catalog (the same stands for test cases, read below), so do not remove this line. You can edit this line to change the test catalog title, anyway.

Just save the page ("Submit Changes") to save your textual changes to the (sub-)catalog.

To delete a Test Catalog you must delete the corresponding Wiki page. Notice that this operation does not delete all of the Test Cases contained in the Catalog. You must either delete each Test Case individually, or first move them into a different catalog first.

If you mistakenly deleted a Test Catalog, you can save the day by recreating a Wiki page with the exact name. To do so, enter the desired name directly in the browser's URL, after the /wiki/ part. Trac will show you an empty page for the catalog, but already populated with all of your existing test cases and sub-catalogs.
Click on "Create this page", give it a title (surrounded by '==') and submit your changes. The new (old) catalog is in place.

Tree View of the Test Catalog

Tabular view of the Test Catalog

Test Cases

Test Cases are the smallest units of test execution.

They are implemented again as wiki pages, with a naming convention that allows the plugin code to recognize them and treat them appropriately.

To add Test Cases into a Test Catalog, open the catalog, go to the bottom of the page and enter a name for the new Test Case to be created into the appropriate text box (blanks and case are supported). Then click the button "Add a Test Case".
A new wiki page is generated, with a naming convention allowing the plugin code to position it correctly in the catalogs tree, and opened for editing.
Be careful that the first line (the one surrounded by '==') will always be taken as the title of the Test Case, so do not remove this line. You can edit this line to change the test case title, anyway.

You can then add the Test §Case description just below the first line (i.e. title), using WikiFormatting, adding attachments and everything else is supported for Wiki pages.

When you are done, save the page ("Submit Changes") to save your new Test Case.

To delete a Test Case you must delete the corresponding Wiki page.

Again, notice the breadcrumbs at the top, useful to go back to the enclosing catalog or any catalog up the hierarchy.

Moving Test Cases from one catalog to another

It is also possible to move a Test Case into a different catalog, with an experience similar to cut&paste.

You first open a test case and click on the "Move the Test Case into another catalog" button. This is similar to a "cut" operation.

You then navigate and open the destination catalog and click on "Move the copied Test Case here" button (which only appears if a Test Case has been cut first).

It is also possible to cancel the operation at any time by clicking the "Cancel" button in a Gmail-type of yellow message at the top of the page.

Test Plans

Since: 1.1.0

A Test Plan represents a plan for a particular execution of all the Test Cases in a Test Catalog (or sub-catalog).

Think for example at the build verification test following a nightly build, or, for traditional projects, Technical Test and eventually Client Test.

Thus a Test Plan is associated to one Test Catalog, or sub-catalog. You can have any number of Test Plans for one Test Catalog, anyway.
The list of Test Plans you generated for a Test Catalog is displayd in a table at the bottom of the same catalog, as shown in the following figure.

To create a Test Plan for a catalog, open the desired Test Catalog (or sub-catalog), enter the name of the new Test Plan in the appropriate test box and click "Generate a new Test Plan".

The new Test Plan will be opened for display, showing all of the Test Cases in the catalog, in the "Untested" status.

As with Test Catalogs, you can choose between tree-like and a tabular representation of the Test Plan, as shown in the figures below.
The former is useful to actually work on the Test Plan and to view test cases before actually going and executing them.
The latter, which indicates for each Test Case the latest status modification and its author, may be useful to keep as the documentation of a just run test plan.

Tree View of the Test Plan

Tabular view of the Test Plan

To track the execution status of a Test Case in a particular Test Plan, open it by clicking on the Test Case name from the Test Plan tree.
Then simply click on the corresponding light in the semaphore at the bottom of the page, as shown in the following figure.

You don't need to save anything, the change is immediately recorded in the database by means of an Ajax call (this API will be documented asap, to allow for setting test case execution status from external applications).

You can customize the outcomes at any moment by adding new outcomes, or modify the descriptions of the current ones.
Do not delete previous outcomes if you have already assigned them to any of your test cases.

For example, to add a new outcome named "It's a Mess!!!" to the failures, add a line like the following, in trac.ini:

red.bigmess = It's a Mess!!!

You will now be able to choose it when assigning a test case status (using the semaphore).
See the following figure.

Linking a Ticket to a Test Case

When viewing a Test Case, you can open a new Ticket by means of the "Open Ticket on this Test Case" button.

The new ticket will contain a link back to the corresponding Test Case and, if you were viewing it in the context of a particular Test Plan, of the Test Plan as well.
Additionally, a programmatic Python API is provided to retrieve all of the Tickets opened against a Test Case, in all or in a specific Test Plan.

This plugin also supports the TracTicketTemplatePlugin to fill a ticket template with this information. In this case, you can use the following parameters in the template to receive the information:

testCaseNumber: The wiki page for the corresponding Test Case

planId: The ID of the Test Plan

planName: The name of the Test Plan

For example, to get the test case number, you template will have something like:

bleah bleah
Test Case: %(testCaseNumber)s
bleah bleah

Showing Tickets related to a Test Case

While browsing a Test Case, either inside or outside of a Test Plan, you can show all of its related Tickets - i.e. Tickets that were opened using the above "Open a Ticket on this Test Case" button.

To do this, just browse the Test Case and click on the "Show Related Tickets" button.

If you are browsing a Test Case "definition", i.e. from a Test Catalog, you will be shown all of the Tickets opened against the Test Case in any Test Plan.

If you are instead browsing a Test Case in a specific Test Plan, you will be shown all of the Tickets opened against the Test Case in that specific Test Plan.

See the following figure for what happens when you click on the "Show Related Tickets" button from the Test Case shown at the previous section.

Custom fields

Since: 1.2.0

Custom fields can be added to the four test objects and to the workflow state object, by declaring them in the trac.ini file.

The syntax is the same used for custom Ticket properties, only the name of the ini file sections are specific: you must use the test artifact type name followed by "-tm_custom".

The test artifacts type names are the following:

testcatalog

testcase

testplan

testcaseinplan

For example, the following sections in the trac.ini file define one custom property for each of the above artifacts.

Once defined in the trac.ini file as above, custom fields will be available to the User for browse and for editing in the Web page, as shown next.

Note: Editing custom properties requires the TEST_MODIFY permission.

The following screenshot shows a custom "platform" field added to the Test Case artifact, and how it is presented to the User for editing.

The value is initially displayed read-only, as a label. Clicking on the pencil icon turns the label into an edit box, allowing the User to edit the value and also displays a "Save" button. Clicking the button immediately saves the new value into the database.

Customizable Workflow

Since: 1.2.0

The following figure shows a sample workflow added to Test Cases with custom sample operations.
No built-in operation is currently implemented but the sample one shown here, named 'sample_operation', which logs a debug message with the text input by the User.

Every object which has a workflow defined is created in a "new" state, so every transition should consider this as the first state in the state machine.

This is a sample content of the trac.ini file to associate a workflow to the Test Case object.

Searching and filtering Test Cases in the tree view

This is available both in the context of a Test Catalog and in Test Plans.

Multiple words (or parts of) separated by blanks are supported, in AND condition.

In the case of a Test Plan, you can also add a test case status to filter by this criterion.
The supported statuses are (even substrings of):

untested

successful

failed

Test Management and Execution Statistics

Since: 1.1.1

Charting capabilities allow for tracking the evolution of the test suites and the corresponding test plans.

To access the test management statistics, click on Test Stats in the Trac toolbar on the upper right corner of the page.

As shown in the next figure, a chart will be displayed with statistics about all the test cases in the system and a default period of time.

By means of the filtering criteria at the bottom of the chart, you can select the desired period of time, the chart resolution (in terms of time between different samples) and the Test Plan for your chart.

You can bookmark the URL named "Static URL" at the bottom of the page in order to go directly with the current selected filtering criteria.

Note: You will need internet connection to be able to display this charts, since it leverages Yahoo remote charting services.

Security

The following new permissions are available to manage the Test Manager security:

TEST_VIEW - Ability to view test catalogs and test cases

TEST_MODIFY - Ability to create and edit test catalogs and test cases

TEST_EXECUTE - Ability to change the status of a test case in a test plan

implement) to differentiate between the definition of the
tests and running a test campaign. An example: I have a
software product, a plugin for Trac. Before doing a release
"1.0" I have to perform some interactive tests. Therefore
I start a new test campaign with svn revision 1234, where
all test verdicts "Untested". Let's say, one or two tests
failed. I conclude the test campaign unsuccessful. Now I
have to fix the bugs and have to start a new test campaign,
again with all test verdicts set to "Untested".

Test objects have been implemented as Trac resources as well, so they will benefit of workflow capabilities.

The new ResourceWorkflowSystem component parses the trac.ini configuration file to find all the workflows defined.
To define a workflow state machine for a particular resource realm, add a "<realm>-resource_workflow" section in trac.ini and describe the state machine with the same syntax as the ConfigurableTicketWorkflow component (I may have borrowed some code here and there ;-).

Custom operation providers can be defined that implement the IWorkflowOperationProvider interface.
They will be asked to provide UI controls to let the User perform the specified operation on the given resource.

This control(s) will be rendered inside a form and the input values will be eventually available to the corresponding provider in the perform_operation method, to actually perform the operation on the resource.

Components that require notification when workflow actions are performed on resources, with or without state transitions, can implement the IWorkflowTransitionListener interface.

Components that wish to augment the state machine at runtime, by allowing or denying each transition based on the object and the current and new states, can implement the IWorkflowTransitionAuthorization interface.

A generic object supporting programmatic definition of its standard fields, declarative definition of custom fields (in trac.ini) and keeping track of change history has been created, by generalizing the base Ticket code.

The database tables still need to be independently defined (right, still :-), but the standard fields can be programmatically provided, and any custom fields can be just declared in trac.ini, with the same syntax as for custom Ticket fields.

Test objects have been implemented as subclasses of this class, so they also benefit these capabilities.

Also, a special sub-class has been created for objects wrapping Wiki pages, i.e. which are based on wiki pages, but with additional fields. Test catalogs and Test cases are as such.

Thus, the following objects have now support for custom properties and for keeping track of change history:

Add the capability to generate a full HTML Test Plan, including the content of all of the test cases. Currently the maximum viewable Test Plan ('Expand All' option) lists no more than the title of each test case.

A full printout of the Test Plan is useful for project documentation and review purposes.

With the addition of custom workflows on any Trac resource, there comes the ability to also define custom operations to be performed along with workflow state transitions.

The TracGenericWorkflow plugin comes with no out-of-the-box operation, and the TestManager plugin only defines a sample operation.

Implement a basic set of custom operations to be used in custom workflows:

"Assign to": only usable with objects that have an "owner" field, allows for assigning an object to some other User.

"Notify": sends a mail to the specified Trac Users, or to a set of specified mail addresses. Optionally allow for specifying the mail subject, and for substituting the object's fields into the subject text.

It would be really useful to be able to take an existing catalog and duplicate it including all sub catalogs and test cases. We could keep our old test cases to one side then allowing our manual testing team to go back and re-run them against older versions of our software.

If there could be a button to fully duplicate a test catalog with a new name it would be great.

It is not possible to add a new Catalog with Trac 0.11. If i try to add a Catalog, the TC wiki page is updated with the new string, but no new Catalog wiki page is created. If i add another Catalog, the Catalog added before is always overwritten. I attached a screenshot (error1.png). The catalog is not clickable or anything else. The wiki index contains only the TC wiki page and no other TC related wiki pages. The database has some entries in testcatalog. All entries have a different id, but always the same page_name (always TC). All other plugin tables seems to be empty.

I can move a testcase into an other (sub)catalog without problems.
But when I try to move the same testcase again into a other (sub)catalog then nothing happens.
No error message occures and the test case does not move.

Trac detected an internal error:
AttributeError: 'NoneType' object has no attribute 'splitlines'
Python Traceback:
File "/usr/local/lib/python2.6/dist-packages/Trac-0.11.5-py2.6.egg/trac/web/main.py", line 444, in _dispatch_request
dispatcher.dispatch(req)
File "/usr/local/lib/python2.6/dist-packages/Trac-0.11.5-py2.6.egg/trac/web/main.py", line 226, in dispatch
data, content_type)
File "/usr/local/lib/python2.6/dist-packages/Trac-0.11.5-py2.6.egg/trac/web/chrome.py", line 737, in render_template
stream |= self._filter_stream(req, method, filename, stream, data)
File "build/bdist.linux-i686/egg/genshi/core.py", line 128, in __or__File "/usr/local/lib/python2.6/dist-packages/Trac-0.11.5-py2.6.egg/trac/web/chrome.py", line 840, in inner
data)
File "build/bdist.linux-x86_64/egg/testmanager/wiki.py", line 76, in filter_streamFile "build/bdist.linux-x86_64/egg/testmanager/wiki.py", line 140, in _catalog_wiki_viewFile "build/bdist.linux-x86_64/egg/testmanager/macros.py", line 81, in expand_macroFile "build/bdist.linux-x86_64/egg/testmanager/macros.py", line 338, in _build_catalog_treeFile "build/bdist.linux-x86_64/egg/testmanager/macros.py", line 650, in _render_subtree_as_tableFile "build/bdist.linux-x86_64/egg/testmanager/macros.py", line 716, in _render_testcases_as_tableFile "/usr/local/lib/python2.6/dist-packages/Trac-0.11.5-py2.6.egg/trac/wiki/formatter.py", line 844, in format
for line in text.splitlines():

Steps to reproduce:
1) Create two catalogs and a sub catalog for each them
2) Create one testcase in each sub catalog
3) at this point switching between "tree view" and "table view" works without problems.
4) now move a testcase from one sub catalog to the other one
5) When you now select the (sub)catalog which was the target for the move and switch to "table view" then the mentioned error occures.

Ticket #7570 has a broader scope than this one, but the present enhancement will allow to easily obtain a list of tickets related to any particular test case, being it in a plan or not.

A "Show Related Tickets" button will be added to the test case and the test case in a plan details pages that will display a query of all the tickes opened with a reference to the particular test case, in any plan or in the specific plan, respectively.

It would be nice to create a testplan with a individual collection of tests. Sometimes it is not useful to test the whole container. The testplan should be something like a template that i can run multiple times.

The TestManagerForTracPlugin uses different Dateformats in the Manager Pages and the Test Stats Pages. The Manager pages are using the local time format of our server where as the Stats pages are using an American Date format. Please use the local server date format also on the Stats pages.

be nice, if the new ticket already defaults to a useful
subject ("Failed test: Basic Sleep - Sleeping Monster").
This can be easily achieved by using the link:
.../newticket?summary=Failed%20test:%20Basic%20Sleep..

When attempting to remove a Test Plan, clicking the X button results in no action and the javascript console showing: "function _ is not defined".

It appears that _("...") style quoting is being used in the testmanager.js file in a few places.

I am not familiar enough with javascript to know if this is supposed to work (I assume it is meant to be for localization?), but adding a dummy function declaration to the top of the .js file fixed this for me:

When trying to install the TestManagerForTracPlugin, the installation hangs in the installation of the testmanager_plugin. I tried to go back in versions to install and found out that release 1.3.11 is the last one working.

The failure I get is, that trac always requests to upgrade the backend. But even after upgrading using the trac-admin command this error isn't solved. So I am unable to start my trac again.

Reporting a mail thread here as a ticket, for reference to other users.

=================================================
Hi Roberto!

I'm sorry to bother you this way directly, but I didn't found anything
useful from google to my problem. I would appreciate if you could help
me a little.

I'm trying to install TestManager 1.4.4 to Trac 0.11.7. I did the
installation from source by easy_install. The installation went ok (I
think). When I try to enable the plugins, trac tell to execute
upgrade. But when executing the upgrade to trac it gives me a error:

Sorry to ask, but my knowledge about python is very limited and this is my first attemt to install a plugin into Trac. I did manage to solve some issues as it stated for example that the TracXMLRPC was missing...

I installed the *.egg files on the admin webpage like shown in ppt-tutorial.
i do not get the request to upgrade environment. If I try to run the upgrade anyway, it says: Database is up to date, no upgrade necessary. The plugins are not displayed on admin page.
I am running trac on windows machine with apache2 phyton 2.5.4 and trac 0.12.1.

first of all , your plugin is very useful and much appreciated. we do have an issue though.

we've installed 1.4.4 and we've started to add existing test cases (from a previous system). we have about 3000 now.
when trying to access the Test Manager tab , it is painfully slow to load ... most of the time, the browser is timing out. Once we're in , navigating the test catalogues and cases is alright.

I should say that the rest of our Trac is working perfectly.

I was wondering if you've seen this before and have any hints for us on how to fix it.

When the testing person is going through the test plan = test cases. It would be great to have a 'Next TC' Button on the bottom

Like after the Change Status you've set it passed and then you would just press 'Next test case' instead of closing the tab and clicking the next case from the plan.

Maybe even so that if the case was last on the (sub)catalog it would prompt to next catalog or at least have the button asking/hinting 'last case, go to next catalog 'XX'?' or be disabled if no more cases on the (sub)catalog

It would be usefull to have some additional Graphics about the Tickets related to a test catalog/plan or multiple test catalogs/plans.

(So I actually present three separated enhancements in one ticket)

As we're using the test plans for each test round we do, it would be great to be able to see the results of a one round as a pie chart.

1)
In a Pie chart It would show in all the configured states for a test plan. for example: Passed, failed, can't test, N/A, etc
This pie could be used to show the result of the round=plan or to show the current state of the testing for that round=plan

2)
A trend line of bugs for a catalog: Three lines: 1) a line for total of found bugs 2) a line of total of closed 3) total of invalid (these are actually the ticket end states)

Of course, this will only work if you keep opening tickets via the appropriate "Open a ticket on this test case" button, in the test case page. This creates a reference between the test case and the ticket, kept in a custom ticket field.

So the idea would be to only create tickets from the test run. so they could be bound to the plan and catalog

The trend line (2) is interesting for a catalog [or multiple catalogs if we end up having smoke tests in separate catalog as the current implementation does not support the possibility to check only certain test cases into a certain test plan. (the other possibility is to name them a bit differently and use the filtering to keep track of the smoke test cases, but that would make the untested status too visible in the end charts)]

3) severity of open bugs
A trend line for a catalog (catalogs) that would show the severities of open tickets. Amounts in vertical, time in horizontal.

From that trend line you could be able to see how long one would have critical tickets open and the amount of critical tickets compared to normal, major, trivial etc.)

Add a third dimension in addition to Test Case and Test Plan, to identify a particular test case instance, for example the platform, or the device.

Which is to say that, inside a single Test Plan, you wouldn't have just one dimension, the Test Case, but also a second dimension, the platform.

So, let's say the use cases would be the following:

Add a Test Variation Dimension (any ideas for a better name?)

The User enters the Admin panel

2) Selects "Test Variation Dimensions"
3) Trac asks for a Dimension Name, defaulted to "Device" (or "Platform")
4) The User can add one or more values for the Dimension, much like it does for ticket statuses, milestones, etc. He can also remove any existing values.

Associate a test catalog to a Test Variation Dimension
1) From a test catalog, the User selects from a listbox the desired "Variation Dimension"

Display the status of test cases in plan, which has a Variation Dimension
1) From a test catalog, the User clicks on the desired Test Plan
2) Being the catalog associated with another dimension, the Test Plan page is in the form of a table, not a tree, with Test Cases in the rows and the associated Dimension values in the columns. In each cell, a red/yellow/green semaphore will display the current test case status. Initially, all test cases will be in the "To be tested" (yellow) status.
3) The User clicks on one of the semaphores and changes the status of a Test Case for the particular dimension value (e.g. platform).

I've already added this on SourceForge, but I'm not sure how often you've time
to look there :-))

When creating a new TestCase, a new Wiki-Page will be opened for editing. At the moment there is only the title added automatically.

What about having a template that will be added to that Wiki-Page (or better said TestCase)? As far as I've seen in the code, this should
not be the biggest problem as the class TestCase has everything that would be needed.

If you like, I could try to provide a patch, but it would be fine to discuss it first... just to be sure to implement it in a way you'd like it :-)
(should it be implemented by a "simple" option in trac.ini? or better implement it
as a "template wiki-page" that will be referenced in trac.ini and then copied when
creating a new TestCase?)

The possiblity to change the statuses already in the tree view without opening the test case would make the testing a lot faster (especially when a slow connection, like for example when the Trac server is located in different continent than where the Testing team is working) if one could change the test result directly from the tree view that is listing the test cases.

This is the case especially when the testers do know the test cases by heart. (Or are using separate test cases tracking and do only report the results into the plugin)

One additional way might be a possibility to set the whole (sub)catalog of test cases into selected status. (However this may lead to a misusage, like first setting all passed and then just setting the one failed -> leading to a wrong status to be marked on the test case status history, unless there would be a timer (for example 15sec) to keep a track of 'mistakes' when setting the statuses and keeping the test case status history 'clean' )

Is there any way to got to a test plan page and print out the information on it a report. that would contain the test case. whether it passed ot not and also the other test plan information. Thank you very much

File "build/bdist.linux-i686/egg/trac/web/main.py", line 511, in _dispatch_request
File "build/bdist.linux-i686/egg/trac/web/main.py", line 192, in dispatch
File "build/bdist.linux-i686/egg/trac/core.py", line 78, in extensions
File "build/bdist.linux-i686/egg/trac/core.py", line 213, in getitem
File "build/bdist.linux-i686/egg/trac/core.py", line 119, in maybe_init
File "build/bdist.linux-i686/egg/testmanager/api.py", line 88, in init

we want to insert a comment at each tescase in combination with the testplan.
our testers want to store information about the test execution (independent if the test faild or was ok) of the actual testplan.
additional there should be at the testcase a link or a box where you see all the comments which where added to different testplans.

I am evaluating Test Manager for Trac to use at my company. We have multiple products each have separate test cases and test plans. I don't currently see a way to separate the test plans by product especially when looking at the reports. It would provide a huge jump in functionality if I could group test plans on some additional criteria.
Thanks! Awesome work.

Currently when I select a testplan with two test cases the pie chart totals at 7 results. This information is easily available already in test activity trend stats. What would be useful to see here, is the number of cases that have last status as success and fail.

I have a problem with the display of statistics. When choosing a Test Stats option displays the following message 'No definition YAHOO' and 'myCChart Is empty and not an object'
I use
1) Python 2.7
2) trac 12.1
3) YUI library 2.9.0
4) Environment without access internet

Trac will link it to wikipage and diesplays "MyCamelCase?", because this link is not existing.
When we add "!" in front of the Name "MyCamelCase" trac will not try to display a link.
But in the tree or table views we will see the "!" - please cut the "!" in front of CamelCase Names.

The current permissions set seems to have a missing permission level. A TEST_ADMIN should exist to encompass all the permissions. QA staff and/or managers would be the likely individuals to receive this permission.

Running version 1.4.8 and Trac 0.12. Catalogs and Test Cases can be created fine with no errors. However whenever a Test Plan is created, the Plan creation works, but after that, anytime I try to view the Catalog that its associated with, I get the following error:

ParseError: malformed start tag: line 1, column 801

I'm currently working around this by commenting out line 247 in wiki.py

first, nice plugin, works rly fine for me and makes my work a lot easier :)

Now for the Problem: If i choose the Stats/Current test Status, with settings set to All Test Plans, somehow the stats sum up the untested test cases, but not the tested ones
For explanation:
I have 1 Test Catalog with 2 Test Plans running(both including all TCs from the catalog). The Catalog includes ~200 Test Cases.
Now the Statistic for all TPs recognizes both TPs use the same TCs, so it says ~200 untested TCs which is absolutely ok for me. But as the test proceeds, i recognized that failed and succeeded tests aren't sumed up, so after both TPs finished, i have like 350 sucessful tested TCs, 50 failed and -200 untested.

I hope my explanations and my english is good enough to bring out the problem :)

I've opened a testplan or a case in plan while logged in. Meanwhile my login has expired and I don't have test_execute permission any longer. Anonymous also doesn't have it. When I try to set test result it seemingly succeeds, orange icon is replaced by green one but the result isn't actually saved.

I installed TracGenericClass 1.1.1, TracGenericWorkflow 1.0.3 and activate them. Then I updated the database and restarted the Apache2. After this I installed TestManager 1.4.9 and activate it. Than i had to upgrade the TRAC Env. There was no Error-Message and the Debug-Log looks good (see attached file). But when I restartet the Apache2 and open the Trac in my browser I have still the message "Database need to be upgraded".
What went wrong?

seems like if the INSTALL.txt file has the correct installation order, it might be just as much typing to include those instructions as a shell script or other method that works just like 99% of the other modules here.. nice project!

We are attempting to upgrade TestManager to the latest version and as a stepping stone, we are going to upgrade from 1.4.6 to 1.4.7 to 1.4.8 to the latest. When we attempt to upgrade from 1.4.6 to 1.4.7, we get the following error:

check the code testman4trac/testman4trac/trunk/testmanager/util.py, line 45,
found CRLF ('\r\n') is used as separator to partition title and description,
but I only found '\n' at the end of line in wiki text, so this partition will always failed, lead description to empty,
use '\n' instead of CRLF can fix this problem.

Looking into this it looks like a common problem that many plug-ins have been having during the 1.0 release cycle. #10230#9742 Looking at the solution used in #10230 I've produced a patch for test manager that at least allows it to install. I'm not sure if this is the right way to deal with this since it looks like #9742 uses a slightly more complex approach.

Looking into this it looks like a common problem that many plug-ins have been having during the 1.0 release cycle. #10230#9742 Looking at the solution used in #10230 I've produced a patch for test manager that at least allows it to install. I'm not sure if this is the right way to deal with this since it looks like #9742 uses a slightly more complex approach.

first I wanna thank you for that nice plugin. We're actually just begining to use it, and had a first look at it. It looks really useful, but there was just this one thing that bothered us.
What we were really missing, was a practical function for the sorting. I know, there is a sorting available through the Admin tab, but sorting via name isn't really satisfactory, as you have to do lots of changes in the names if you insert a new case for a new feature like somewhere in the middle. (As you have to use numbers to sort the cases the way you want them)
Our idea was, maybe it's possible to add a sorting via some kind of internal id, maybe with "sort up/down" buttons which pushes the case/folder higher/lower in the sortposition.

On the Template Admin page a long list of non existing test catalogs without names is displayed. The only catalogs I really have are the ones that have names. How can I get rid of the zombie catalogs in this list? I'm running Test Manager 1.5.2 on Trac 1.0.1.

On the Template Admin page a long list of non existing test catalogs without names is displayed. The only catalogs I really have are the ones that have names. How can I get rid of the zombie catalogs in this list? I'm running Test Manager 1.5.2 on Trac 1.0.1.

Trying to delete an unassigned test case template in the test manager template admin page results in a warning, that the template can't be deleted as it is in use by a test catalog. The warning appears also for templates that are not assigned to any catalog at all.

I just had another idea for an enhancement.
The idea was to add a combobox to choose a component (related to components for tickets) to the edit page of test cases / sub-cataloges.
Would be really nice, so when you open a ticket to a test case you got that work done already ;)

Our Trac runs on a server with US locale and we have set the date format to ISO 8601 as we all hate the stupid American date format. Throughout the whole Trac installation we therefore see and can enter dates in the format YYYY-MM-DD except for the Test Stats page. The two date entry fields only accept the American date format MM/DD/YY and don't feature the new Trac wide date picker. Please allow and display dates in the format selected in the Trac Admin page and/or include the date picker.

We are using Test Manager 1.4.10 and we're encountering an intermittent problem with custom fields not showing up in test case pages within test plans. We would try refreshing the page and the custom fields would either be displayed or not with no consistency. We are using IE 8 and Chrome version 22 browsers.
Attached are images that show the HTML portions that should have been generated when it works and what HTML was missing when it fails.

With Trac 0.12 and PostgreSQL there is a datatype issue. For example when trying to create a TestPlan Pg won't store it because the value for the "time" column does not fit into the integer type range.

As a work-around I changed the data type on my local instance using "alter table testplan alter column time type bigint;" but I guess resorting to bigint is not the nicest resolution.

Sometimes we only small test runs with only some parts of the catalog. In a such an test plan we mentioned that sub catalogs without test cases are displayed, too. This leads to an messed up test plan. I think it wouldn't hurt someone when this sub catalogs aren't visible.

I'd like to have a plain clone of the test plan, with all it's tests set to status "to be tested".
I'd like to be able to modify the name of the test plan, so that it does not state "v1.0 test plan (copy)" ,
but what I want it to say, e.g. "v1.2 test plan".

First of all, big thanks for your great job !!! ;-)
I wonder how to change the template of the ticket ?
In fact i would like to set automatically some fields when using the open ticket feature from a test cases.

I was trying to install testmanagerfortracplugin and everything looks fine while the installation. But it occurs an error when I open the "Test Stats" or "Test Manager" tab, because the system was not able to create the database tables. I have no clue what I did wrong. I installed the egg files in the correct order and restart the server every time. In the description you mention that a database upgrade command will be displayed, but the system never displayed me something like that. I finally execute the standard trac-admin upgrade command, but nothing changed.

The Stats don't appear because the versions of JQuery and JQuery-UI have been updated (1.9.1 and 1.10.0 respectively), where the links in the page haven't and are pointing to versions 1.5.1 and 1.8.13 respectively.

Only all of the test planes are shown in the charts [ pie, trend ], changing the test plan has no effect on the charts but it does in the url static link. For us this is critical as we cannot get a snapshot of the current state of the plan in terms of test executions. Thanks in advance.

I am using Trac 1.1.2dev-r11803 with MySQL and trying to install TestManager. The two generic plugins install fine, but when I install the actual TestManager plugin and upgrade Trac afterwards, I get the following error:
OperationalError: (1071, 'Specified key was too long; max key length is 767 bytes')
trac.log says it is the "CREATE TABLE testcase" that fails I can reproduce the error in MySQL, and I can create the table if I ommit the PRIMARY KEY part.
I created the Trac database with this:
CREATE DATABASE trac_myproject DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;

As far as I can see in the Trac documentation for MySQL, this should be all right, so what can I do to make the plugin work?

If to create a test plan and then add a test case inside of it, then click on a test case and choose "Open a Ticket on this Test Case", then test case name (TC_TT4_TC11) goes to the field "Test Case:" on the bottom of a ticket and test plan number (9) goes to the field "Test Plan:".

If then click "Show Related Tickets" there won't be any info about this ticket, because query looks like ​this instead of ​that

When a test plan is created and only some of the test cases are selected, the test plan still shows the whole catalog hierarchy, even if some of them are empty.

Can this be set not to show the hierarchy at all? Or at least not to show the empty ones, because their status can not be changed to success since they do not contain cases.

In the example below I selected 8 (out of many) test cases in system basic and 1 from unit tests. The plan has to be in this part of hierarchy because it contains cases from multiple catalogs, but the empty ones should not be shown (and have statuses)

In our company, the quality system requires the test report. The test reports contains the list of test with results (pass, failed) and the exact version tested. This is what I really missing on TestManager Plugin. it is blocker ticket for me. I cannot use it unless I am able to proove to quality what exactly was tested, what exactly failed, what exactly passed.

Let assume, that the next release version is 2.x. I create test plan with name complete tests for 2.x release. The testing department is testing the new version for 2 months and in this times several bugs needs to be fixed, so the are several version being tested: 2.0.1, 2.0.2 and 2.0.3. It would be really helpfull to link the result of test with the tested version. I suppose, that custom fields can be defined only to test cases not to test results. Am I right?

Here is the patch I had to do for this plugin to work in Google Chrome browser.

In /usr/share/trac/myproject/plugins/TestManager-1.7.3-py2.7.egg
I opened it with Archive manager (GUI) and edited /usr/share/trac/myproject/plugins/TestManager-1.7.3-py2.7.egg/htdocs/js/testmanager.js.

I had to replace $("input[@name=' with $("input[name=' removing the @ symbol to avoid a jquery error. This was done in about 2-3 places in this js file.

Custom testcase (or testcase in plan) fields are not being returned with XMLRPC Calls. Since the base class AbstractVariableFieldsObject already has them it seems sensible to return them. Example for Testcase in plan:

Original return value for a tescase with getTestCase when plan id is provided in rpcsupport.py

The current incarnation of the custom fields for a testcase (or testcase in plan) is restricted to text and they are kept in a separate table in the database. (Although I imagine the plan is to extend the capabilities and allow other types)

One possibility would be to have custom fields as Wiki Pages themselves allowing for full wiki formatting of the field. Use case:

Test case has the standard "title" and "description".

Custom fields "Execution Steps" and "Expected Results"

The latter are calling for full blown editing and quite probably with Wiki Formatting. The current "text field" presented in the web browser is rather limited as it won't display even line breaks.

The already built-in mechanism for versioning using the wiki pages would be re-used.

To avoid compatibility problems with existing deployments a possible path would be to still allow the os.type = text and also allow os.type = wiki.

I noticed while quickly trying out your plugin, that version 1.8.1 of the source has mixed tabs and spaces, which can cause ​problems. There are just a few tabs, mostly around import statements, but also dispersed throughout the files.

I have a generic Test Case template which most of the Test Catalogs use.

Every time I add a catalog, I need to go to the admin panel and assign this template manually.

It would be nice if there was a "Default Test Case Template" that is automatically assigned to all newly created catalogs by default.
This template should be empty in the beginning (so it behaves like no template is assigns), but can be editable like a normal template

Following situation:
We created a TestCases and copied this two or more. After coping we change the content of the TestCases. After that we created a Testplan and executed this. If we want to create a Ticket via the Button in the TestCase, it doesn´t work. The Button did nothing.

I can create Catalogs and Test Cases just fine. But when in "tree view" I see nothing when I select a Test Plan.

When set to "table view" - I do see a table, but I can't usefully interact with the Test Cases.

There is a screenshot in the description where a Test Plan has custom states / settings, and someone is using a drop-down list to change the state of a specific Case within a Test Plan. That's the functionality I need! I'm so frustrated to be so close...

I'm using TestManagerForTracPlugin with PostgreSQL database. When I try cloning a test plan an error is reported. The reason is the INSERT in testplan table. This table has two fields defines as INTEGER:

Hi,
I have installed the latest trac plugin for test manager. I could add test cases by creating test catalogs and I have also created test plan. The only problem I am facing right now is I could not view the test status in 'Test Stats' tab. Please refer to the attachment. Please help me in getting rid of this issue.

As we are using a Theme which has a sidebar with the file hierarchy, I would like to know if it is possible to configure TestManager in order to store the Wiki-pages not under the wiki-root, but under subfolder of wiki-root.

I'm considering using the TracGenericWorkflowPlugin in a plugin that I maintain, however the GPL license presents a problem. The GPL license of TracGenericWorkflowPlugin will require plugins utilizing TracGenericWorkflowPlugin to also be GPL licensed. However, the plugin I maintain uses the less-restrictive BSD license, same as Trac. Would you consider changing the license to 3-Clause BSD license? That would also open the possibility of eventually integrating this code into the Trac core.

Download

Source

Installation

Both Trac 0.11 and 0.12 are supported, and also both Python 2.5 and 2.6.

The plugin can be installed as usual, from the Trac administration panel, and requires a database upgrade.

You can follow the instructions in the INSTALL.txt file contained in the binary packages.

The functionalities are divided in three plugins, which must be installed in this order:

1) Trac Generic Class

2) Trac Generic Workflow

3) Test Manager

An additional plugin is only useful for debugging and should not be installed in a production environment.

SQL Executor

With the latter plugin, you can perform arbitrary SQL queries on the Trac database from a web browser.
Just type on the URL:

http://yourserver/yourenvironment/sqlexec?sql=any SQL statement

For example:

http://localhost/sampleproject/sqlexec?sql=select * from testcatalog

Follow the tutorial presentation that you can download from the project site to start.
The tutorial is actually very old and needs to be updated to the new Test Plans and other features, but it will get you started.

Recent Changes

o Fixed Ticket #8378 (Track-Hacks): Set date and time format correctly in Test Stats page
Also added support for custom test case outcomes in the Test Stats page
Added Wiki documentation on the ability to customize the test case outcomes (i.e. states, verdicts) and to link Tickets to Test Cases.