17 September 2016

Risks of Testability

What potential problems should you be willing to risk in order to make your application more testable? A testable app is one that allows you to question the application in anyway required and to get the answers to those questions quickly.
A story.
We were creating an iPad application and we decided to run some tests through the application user interface. Perhaps the main issue with UI test runners is the speed. With browser tests we are usually able to shortcut to the bit of the application we are interested in, generally by using a URL. We couldn't find an equivalent way of doing this in app without introducing code that is test specific (I added configuration flags that allowed skipping certain views, like log in). One of the developers didn't like this.
We discussed it and agreed to remove it. On the proviso that it would be looked into further and if a better way could be found we would use it, otherwise, we would revert (which was the first mistake on my part).
Shockingly, it never got revisited. Every time I brought it up it was "We'll look into it as soon as the work on this feature is complete" and I let it slide (which was my second mistake). I couldn't get them to prioritise testability over features, because despite what is said at the start of a project about the 'ilities', when all the PO sees is a list of features required for MVP to be completed, those 'ilities' are never prioritised (which is a rant for another day).
So we ended up getting stuck with a test suite that, every test, would take at least 6 seconds (log in uses a six digit pin, and the UI Automation framework could only 'press' a button a second) to get to the bit you are interested in. This meant every test we added put at least six seconds onto the run time of the suite. It led to reluctance to add anything as it wouldn't provide fast enough feedback. It led to the tests not being run locally, because of the time it would take.
Now here is the question. Is the risk of a user being able to get the app into an unusable state because of our testability code greater than the risk of missing issues because it is too painful to test?