Request for Comments - Module::Install support for xt

With the increasing usage of the xt directory, and with a more robust understanding of the appropriate mechanics behind release tests, I feel the time has come to build support for xt into Module::Install by default.

But before I do so, I'd like to lay out the rules I'll be applying to the implementation for public review and commentary.

-------------------------------------------------------------------

1. The xt directory, if it exists, will automatically be added to the list of tests to be executed if the Makefile.PL is running with either..

1.1. Author mode enabled

1.2. With RELEASE_TESTING true.

1.3. With AUTOMATED_TESTING true.

2. Recursion on the xt directory will be triggered with the same tests_recursive command that triggers recursion on the t directory.

3. The behaviour of the individual test scripts in xt is assumed to be...

3.1. Always run when RELEASE_TESTING is enabled.

3.2. Run during AUTOMATED_TESTING only if the support modules are installed and at a new enough version.

3.3. Run during author mode only if the support modules are installed and at a new enough version. If the test script is unable to M:I's author mode status, the test should be skipped to prevent accidental executing during end user install.

I believe this combination of rules will give us the following scenarios.

Module Authors

Having checked out a module the xt tests will be visible during make test and will run if they can, but if not they will skip without error while telling the author what extra dependencies they need to install.

When building a release tarball, the xt tests will forcefully always run and fail if they are missing dependencies. This will ensure releases are always subject to full Quality Assurance.

CPAN Testers

The xt tests will be included in the list of tests to run, but will only be run when the system already has the appropriate dependencies in place to prevent excessive dependency recursion (which might fail and prevent any testing of the package at all).

End User

During installation, the end user will never see the xt tests displayed on their screen, even if they would otherwise always skip.

-------------------------------------------------------------------

If you have any comments, questions, or suggestions, please reply now.

Assuming no obvious issues arise, my plan is to implement this in the next M:I release some time in the next month.

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Without JavaScript enabled, you might want to
use the classic discussion system instead. If you login, you can remember this preference.

Please Log In to Continue

I generally think of CPAN Testers as trying to reflect the "end user experience" -- and some people just happen to report in bulk using automated smokers. I would rather just have xt tests skipped if AUTOMATED_TESTING is true, since an xt FAIL is meaningless to an end user.