Main menu

Today testing.drupal.org began sending results back to drupal.org! The results are displayed under the file they relate to. The reporting will not trigger a change in issue status if the testing fails, as is the future plan. That will be turned on when the automated testing framework has proven to be stable.

The following is an example of what a pass result look like on drupal.org. (http://drupal.org/node/180379#comment-739254)

The following is an example of what a fail result look like on drupal.org. (http://drupal.org/node/200185#comment-752032)

Early success
Before the actual reporting was turned on the automated testing framework discovered two core issues. One was a partial commit that caused a PHP syntax error. The other was a bug with the installer. I hope to see the results make patch reviewing much faster and easier.

Tags:

After several days of battling the server and the enormous XML file generated by exporting the assertion data a solution was found. Instead of dumping all the results the framework only stores summary results. Just enough to tell you where the tests failed, but at the same time preserving the server.

Intended workflow
After a developer notices that there patch unexpectedly failed they will run the failed tests on their dev environment to find out what went wrong and work from there. This keeps the debugging work in the developers environment instead of on testing.drupal.org.

Patch criteria
One of the remaining issues pertains to what patches should be tested. The following represents what I believe we need from reading the issues and pondering. Some of these are already implemented, but listed for clarity.

File must have patch or diff extension.

Issue is for Drupal 7.

Issue is marked as patch (code needs review) or patch (reviewed & tested by the community).

Issue is not in documentation component.

Do not test flag. Something in filename like D6_*.patch if in Drupal 7 issue, like a port. And possible NOTEST_*.patch or a component? This is the part that needs to be figured out.

If the patch meets the criteria then it will be eligible to be tested and at some point sent for testing.

Patch sending event
Patches need to be sent out frequently to ensure they are tested as quickly as possible. To do this drupal.org will be setup to send files to be tested every 5 minutes.

In addition once daily, every other day, or some similar pattern the entire issue queue should be searched for patches that match the above mentioned criteria, regardless if they have already been tested, and send them all for testing. This will ensure that "patch rot" or patches that no longer apply after other changes are committed are automatically detected.

Reporting back
The most exciting and useful feature of this entire endeavor is being saved until last. Once we have all the things mentioned above in place and testing.drupal.org running smoothly then reporting back will be turned on.

What reporting will do is after a patch has been tested the results will be sent back to drupal.org. Code running on drupal.org will then check the issue related to the file results sent back to see if there is a newer patch. If there is the results will be ignored. If not then the issue status may be changed to reflect the result of testing.

In addition I envision having the results for each file listed on their comment in a summarized form with a link to the full results on testing.drupal.org.

Roadmap
In order to make this a reality a standard needs to be decided on for the patch criteria, either confirming/appending to the above, or something new. Once that is complete then the code needs to be written and placed on drupal.org.

After that is complete testing.drupal.org should be monitored to ensure it works properly and consistently. In the meantime work will begin on the code related to reporting back to drupal.org.

Community help
It would be very much appreciated if the community would check the results of testing on their patches to Drupal 7 core. This can be done be visiting the following page.

http://testing.drupal.org/pifr/node/1/[node_id]

Make sure that the testing results work and that your patch is tested if it meets the criteria. Please check even if your patch should not be tested, to ensure that it isn't.

Tags:

I recently had a talk with hunmonk in IRC about what needs to be done to get drupal.org to start sending patches to the new automated testing framework. After almost an hour of discussion and thought we came to the following agreement. (summarized from IRC log)

Initial plan

PIFT which is already in place on drupal.org will continue to be used to scan the issue queue and send patches to testing.drupal.org. (May need criteria updated.)

PIFT client will then be ported to Drupal 6.x and benefit from the addition of a hook as apposed to the current mechanism.

PIFR will be updated to implement the new PIFT hook in the ported Drupal 6.x client.

Future plan

Reporting back to drupal.org will be discussed a bit further as there are several ways and opinions on how it should work both technically and from a workflow standpoint.

PIFR will send summary of testing results back to drupal.org through PIFT.

Tags:

The idea of automated testing using SimpleTest has been around almost as long as the SimpleTest module itself. A movement to accomplishing this has been centered at testing.drupal.org and the following projects:

Testing.drupal.org

The goal is to have all patches against Drupal 7 core, over 1000 currently, automatically tested to ensure that they apply and pass the suite of tests which consists of over 5500 individual assertsions for about 70% code coverage. Completing this goal has been long in the making by a number of individuals, who have been focused on Drupal 4.7, 5, and 6, but now appears to be realized. Being heavily involved in SimpleTest development, I have taken on the task of finishing up the automated testing framework to be compatible with Drupal 7 and SimpleTest core.

Hurdles
The automated testing framework was designed to support Drupal 4-7 through a 5.x module. This approach worked well, but with the recent explosion in SimpleTest 7.x development the workflow and design has been changed.

The XML reporter is no longer available, and is necessary for transferring results from test slave to testing master server.

The Shell script to run tests does not work properly in that it returns different results than the web interface.

The Drupal 7.x SimpleTest framework does not support META refresh and thus cannot run batch API pages necessary for running the tests and installing.

The Drupal 7 installer does not auto detect JavaScript support and does not work without JavaScript. (issue)

The XML output, over 2.7MB, is too large to transfer between servers and will only continue to grow with more tests.

Goal
The desired workflow is described below.

Steps

A file is attached to an issue at drupal.org.

The system determines if the file is a patch/diff and fits the proper criteria.

The file and issue information is sent to testing.drupal.org.

The file is added to the testing queue.

When a testing slave is available the next file in the queue is sent to be tested.

Once the slave has completed the testing the result is sent back to testing.drupal.org.

The result is then store and then the next file in the queue is sent to the slave.

This will provide a very consistent way to test files that will allow for scaling in the event that additional servers are necessary to deal with the patch load.

Solution
The previous framework is described at http://testing.drupal.org. The new framework is based heavily off the old framework, but runs in Drupal 6, has been re-written, and is better suited for running tests against Drupal 7.

The re-written code resides as Project Issue File Review. The single module can act as any combination of the three roles: project server, testing master server, and testing slave. That means that the current single server that testing.drupal.org has available can act as both the testing master server and a testing slave. This setup can encompass internal business testing on a single server as well.

The module addresses the hurdles in several ways.

A patch is applied to Drupal 7 HEAD that converts the output from the standard HTML interface to an XML dump.

Instead of using the shell script the code uses SimpleTest with a patch that adds support for meta refresh.

When installing Drupal 7 HEAD the batch API is forced to use META refresh.

The XML output is gzipped to allow for it to transfered between slave and testing master server.

Advantages
Some may wonder why I re-wrote the existing code, there are several reasons.

Drupal.org is being redesigned in which it will also be ported to Drupal 6 so the testing framework might as well lead the way and not become a Drupal 5 dependency.

Solution that is tailored towards the SimpleTest 7.x workflow.

The previous framework kept each test installation around. Since SimpleTest no longer works on the active database all that would be seen is a base Drupal installation and is thus useless to keep around.

Removes SimpleTest library dependency.

Removes SimpleTest module download dependency during Drupal HEAD installation as it is now in core.

Code logic built around the community's desires for testing.drupal.org. (explained further bellow)

File results are displayed in relation to the node they originated from which makes for easy relation between drupal.org and testing.drupal.org. As apposed to the previous system that assigned a random ID to each file tested. This setup still allows for multiple project servers with the same issue node IDs.

Files are sent for review immediately after being posted. (further explained below)

I much prefer working in Drupal 6 and when going through the trouble of porting why not do a bit or clean-up/re-write.

What to expectSummary of files per node
To view a list of all the files reviewed for an individual node you would enter a URL like: http://testing.drupal.org/pifr/view/1/9 where 1 is the project server ID, could be replaced by textual "drupal.org", and 9 is the issue node ID. All the files that testing.drupal.org is aware of will be displayed along with their current status and testing result. The possible status values are: Queued, Sent to slave #, and Reviewed. This allows developers to check on a files testing status at any time. Although not displayed the comment ID is also stored for future auto commenting back on drupal.org issues.

Detailed test resultsUpon clicking "View results" the detailed assertions will be displayed from the tests run. Currently this consists of a dump by test case. In the future this will hopefully look more the like the standard SimpleTest interface in Drupal 7. That will be handled in core when the test results layer is abstracted. I have already created a patch to begin this with the separation of test selection and test results. Once completed I will attempt to further cleanup the results display to work nicely for testing.drupal.org and any other modules that wish to integrate with SimpleTest. That will also allow this results page to be easily updated with the latest improvements to the SimpleTest results display.

Status summaryIn addition to checking the status on individual files the overall status of the queue and testing slaves can be viewed by entering the URL: http://testing.drupal.org/pifr/status. The operations will only be visible to administrators. This will give an a general feel for the load being put on the testing infrastructure and allow for easy reset of a slave server when they crashes.

What is left
There are several things left to be done, some of which require the community's involvement.

Drupal 7 HEAD does not pass all tests. That means that when testing.drupal.org is turned on all patches will fail. That defeats the purpose of testing.drupal.org to some degree. Anyone can help to fixing the tests or issues related to the failures. Sadly it used to have 100% passes.

Testing results code needs to be abstracted once the clean-up of backend code gets committed.

Determine the proper criteria and workflow for choosing what patches to test and how to report back. (more detail below)

Code review would be nice, but lets focus on critical issues and worry about minors ones once testing.drupal.org is up and running.

Patch criteria and workflow
This is the one area that needs to be flushed out. As many may have noticed when testing.drupal.org was turned on to simply test if patch applied to the current development branch, refereed to as HEAD, it created a number of issues. The following are issues that need to be resolved.

When should patches be sent to testing.drupal.org from drupal.org: right away or wait for cron?

If newer patch is posted when test results are returned should the auto comment still be sent?

Should the workflow be amended to have something like patch(tested) between patch(needs review) and patch(review & tested by the community).

One advantage with sending patches right when they are posted is that the logic about when to comment back can be put on testing.drupal.org as opposed to having a shared more complex logic on both drupal.org and testing.drupal.org. Since all that is sent is the node ID, comment ID, and file URL it shouldn't be a load on drupal.org.

Comments are appreciated, but keep in mind that the code is still being developed and there are a few things I will add before attempting to install on testing.drupal.org. Part of which is a Drupal 5 piece of code to send files, I might just modify the previous framework for that.

Tags:

For those of you who don't follow Drupal 7 development, the SimpleTest module was committed to core. I have been maintaining the both core and contrib versions of the module and decided to back-port the enhancements made in core to contrib.

Because of the back-ported 2.x version you can write your tests in the Drupal 7 format that is the future. It is highly recommenced that you convert tests to this new format as Drupal 7 will not support it and Drupal 5 and 6 may not support the old format much longer. Converting will also help because the documentation available is being converted to the new format and many new features are available.

If you are interested in starting to write tests or getting up to speed on recent development I have create a getting started document that will explains the different modules available and give you a brief history or changes.

If anyone is interested in helping me back-port the changes to 5.x that would be great. Better yet just get it working and submit a patch.

Tags:

I have released the second alpha of the Usability Testing Suite. The API is fully functional and there are two plug-ins implementing the API. The reason the module is still marked as alpha is due to the user flow issues and clean-up items.

Tags:

The latests and greatest version of openSUSE has been released. I have been waiting for SUSE 11.0 for some time and will be upgrading immediately. I have already backed up my system and have begun downloading the 64 bit DVD via torrent.

This release has been praised as the Mercedes-Benz of Linux. The release has lots of updated programs, new features, and performance upgrades.