The only reason this is working with the current resource adapter is because the resource adapter is checking for the status on the userTransaction object on every operation. But the TransactionManager should call the proper end here with the abort status IMO.

This could be replicated by the test org.jboss.test.jca.test.TransactionActiveUnitTestCase on the HornetQ integration branch on the application server's SVN.

Whilst it's not optimal it's not a bug either. The flag to end refers to the work on the branch rather than to the tx outcome. The sequence end(success); rollback(); is valid and will occur not only from the reaper timeout but also in a normal tx termination where a prepare on another resource fails and the tm executes a rollback. Therefore, whilst we could change the reaper (and the normal rollback path, which is largely the same) to use end(fail), your code is going to have to cope with end(success); rollback() anyhow.