In March, we exhibited at the 33rd SQA Annual Meeting and Quality College and spoke with many people from regulated industries who tested web applications using paper based manual test. Almost uniformly people expressed frustration with paper based testing, especially the amount of time it takes. One woman in particular had just finished testing a web application with paper tests that took her five months to execute. In our experience, it is common even for teams of five to ten people to take weeks or months to finish a suite of paper tests.

Paper based web application testing takes too long to implement, and our desire to speed up this process is one of the factors that lead us to create Toffee. In our prior blog post, Manual Testing: Toffee is about Orchestration not only Automation, we explained how Toffee improves the speed and accuracy of manual testing by eliminating paper and promoting heads up testing. With Toffee, testers no longer need to switch contexts between the application under test, paper test scripts, and a text document for tracking screenshots. Now, we want to share an experiment we performed to test the time efficiency gains of using Toffee tests versus traditional paper based tests.

As part of our experiment, we created two sets of tests for the exact same functionality of a food ordering web application. One set of tests was a pure paper based test consisting of 47 manual steps. The second set of tests was Toffee based and although 7 of the 47 steps were automated, the rest were still manual.

Twenty different people executed both sets of tests with half doing the paper based test first and half starting with the Toffee based test. We timed how long it took each person to complete each test and found that the Toffee test took 31% less time to complete. So, even though 85% of the Toffee test required the exact same interactions with the application under test as the paper test, people were able to complete the Toffee test in about two thirds the time it took to perform the paper test.

This is a huge gain in time efficiency especially if tests take weeks or months to complete and just one of the many reasons to use Toffee to orchestrate your testing.

For more information about Toffee, visit http://toffeetesting.io, or feel free to contact us with questions at info@toffeetesting.io.

How do you properly test the interaction between multiple users within a web application? What if you need to test that user B receives a notification immediately after user A performs some action? How do you ensure that user D’s access is cut off immediately when user C revokes it? Testing this type of functionality can be difficult, because it is not as simple as opening the application under test and typing in fields and clicking buttons. In order to ensure that the interaction functions properly and timely, you often need to concurrently open, interact with, and monitor at least two unique instances of the application.

Consider a chat app that allows two users to instantly send messages to each other through your web interface. To test this application, you must log two different users into their own instance of the app at the same time. To test that the messages are instantaneous, you need to send a message from the first user and immediately check if the message appears in the second user’s inbox. You cannot simply log in to the app with one user, send a message, and then log in as the receiving user because the immediacy of the message exchange is not tested.

Collecting screenshot evidence of user interaction adds another layer of complexity because, you have to position each unique user’s instance of the application on the screen so that they do not overlap with each other.

Manually testing user interaction quickly becomes time consuming and complicated as you need to test and gather evidence for more unique users. Many web testing tools do not even offer the ability to simultaneously run multiple instances of the application under test, however, Toffee has features to make it easy including multiple test sessions and comparative screenshots.

A Toffee test session is a single active instance of a browser that is then used to access the application under test. Toffee allows you to open multiple test sessions at the same time. This can be used to simulate multiple users because each test session can access the application as a different user. As in the chat app example, Toffee can log in to two instances of the app, send messages from one user, and confirm that the messages are immediately received by the second user.

Toffee makes it simple to document the user interaction with comparative screenshots. Using Toffee dock or move commands you can easily set up a comparative screenshot so that each unique test session does not overlap. Dock commands place a session at either the top, bottom, right, left, or any corners of the screen. Move commands allow the user to set a session to any user specified screen coordinates.

Testing multi-user interaction can be a complicated nuisance, but Toffee’s test sessions and comparative screenshots make it simple.

For more information about Toffee, visit http://toffeetesting.io, or feel free to contact us with questions at info@toffeetesting.io.

As part of Leotta’s study, testers with some programming experience made two sets of tests for six different web applications. Each web application had one set of tests made using a record/replay tool called Selenium IDE and another set of programmed tests written in Java using Selenium WebDriver. The testers then updated each test suite for a newer version of each web application. Leotta measured the time the testers spent creating and updating the test suites, comparing the record/replay time with the programmable test time.

Leotta found that the programmable tests took up to 112% more time to make from scratch than the record/replay tests. This was mainly because creating the record/replay tests did not require any programming knowledge and was as simple as the tester recording themselves using the web application.

Next, Leotta’s research showed that when test suites were updated for new versions the programming tests took up to 51% less time to repair than the record/repay tests. This time saving was mostly due to the developers using the page object design pattern. In summary, a page object is a programming construct where code that locates elements and initiates actions on the web application is stored in one place. The page objects can be reused in multiple tests. Testers only update page objects when repairing programmable tests, which can fix multiple tests at once. Record/replay tests do not reuse anything, so every record/replay test must be fixed individually or even recorded again every time the application under test changes.

As I discussed in a previous post, Toffee avoids the brittleness common to record/replay testing, replacing it with plain-language test commands and an interactive script development environment. Toffee also saves time during test repairs with aliases and test modularization, which provide the same benefits as the page object pattern, but without requiring programming. Toffee commands that are repeated throughout tests can be saved as an alias, so only the alias needs to be updated. Also, a series of frequently used commands (think login) can be saved and maintained in a single “include” file and used on multiple tests.

If the application will be updated more than once, Leotta recommends using programmable tests, otherwise, he recommends record/replay tests. Toffee has the efficiencies of both record/replay tests and programmable tests, so it is the best solution for whatever your web testing needs may be.

For more information about Toffee, visit http://toffeetesting.io, or feel free to contact us with questions at info@toffeetesting.io.

Web application testing for regulated industries often involves auditors, who are not developers, to confirm the testing results. We developed Toffee so that non-developers can read the automated test results and quickly understand them.

We have personal experience of the challenges of incorporating automated testing into the computer system validation process for regulated industries. Before we built Toffee, we wrote our automated tests in Java using Selenium. After writing a large suite of automated tests that validated one of our client’s applications, we turned over the results to their auditor. The auditor came back to us and explained that our results did not help her because, understandably, she did not know what any of the Java meant. We ended up sitting down with her and explaining each line of code.

It quickly became apparent that describing lines of code was not a tenable exercise for regulated industry web testing. We think auditors should not have to become developers, so instead we made Toffee to allow auditors or any non-developer to read automated tests.

After Toffee runs a test, it generates a report that conforms to the traditional web testing style used in manual/paper tests. The report has five columns for each test step which state the action taken, the expected result, the actual result, the verdict, and if applicable the screenshot evidence.

If the test uses screenshot evidence, a picture is taken for each step. We discussed this in a prior post called Automatic Screenshots: Evidence Gathering Made Easy. Each screenshot is included in the report appendix with a reference number, so all the information is in one place. This makes it easy for an auditor to quickly evaluate the test.

We realized that non-developers are regularly a part of web testing, so we made Toffee to be user friendly for everyone regardless of their programming knowledge.

For more information about Toffee, visit http://toffeetesting.io, or feel free to contact us with questions at info@toffeetesting.io.

The convenience of using record/replay web testing tools is often defeated by the brittleness of the tests they produce. Mouna Hammoudi’s paper “Why Do Record/Replay Tests of Web Applications Break?” explores why these tests break so easily as the application under test changes. Toffee overcomes many of these issues with record/replay testing.

Hammoudi used a record/replay tool called Selenium IDE, which allows a user to record themselves using the application under test and replay their actions at a later time to test functionality. After creating a test for each of 300 releases across 5 web applications, Hammoudi found that roughly 73% of all breaks in her Record/Replay tests were due to locators.

Hammoudi explains that locator breaks occur when either an HTML element’s attributes or the HTML structure has changed. In our experience, tests that use structure based locators such as XPath are especially vulnerable: as web applications are updated the structure changes regularly and even small changes can invalidate structural locators.

Toffee offers two solutions that make Toffee tests more robust and less expensive to maintain compared to record/replay tests. First, Toffee makes it easy to write test steps using attribute based locators like id, name, label, and others. These locators tend to be more stable because they are not affected by common structural changes. Instead, developers have to intentionally change the attribute value.

Second, Toffee provides users with the ability to create aliases. Aliases are custom commands which are defined by the the tester and refer to other Toffee commands. For example, the command “click button with id button-logout-user” can be aliased as “logout”. Then the “logout” alias can be used throughout your tests, instead of the original command. This minimizes the necessary repair after the button’s id changes because only the “logout” alias must be updated.

The drawback of record/replay tests is their brittleness. Toffee tests are more robust and reusable without sacrificing the accessibility of record/replay tools.

For more information about Toffee, visit http://toffeetesting.io, or feel free to contact us with questions at info@toffeetesting.io.

We created Toffee to not simply automate QA testing but to orchestrate both manual and automated tests. Manual tests are an important part of QA testing because it is either impractical or impossible to automate everything. See our prior post on the virtues of manual testing: Two Cheers for Manual Testing

Creating manual tests for Toffee is as straightforward as writing traditional manual tests. Simply mark the test step as manual and then write the instructions for the tester to perform. Toffee will know it is a manual step and will take care of the rest.

Manual test are typically carried out using stacks of paper or lengthy text documents with instructions and a space to fill in the result. Testers are stymied by going back and forth between the application under test and the test documentation, squandering time and becoming susceptible to mistakes every time they lose their place in the instructions.

Toffee eliminates these problems. Testers no longer have to deal with shuffling paper or scrolling through online documents. Toffee provides test step instructions, and a place to record the results for only the current step, in a “heads up” display alongside the application under test. This reduces human errors and is less taxing on testers.

As part of testing orchestration, Toffee allows manual steps to be integrated directly into otherwise automated tests. Toffee executes all of the automated steps until it comes to a manual step, then it will pause and request the manual action from the tester. After the tester completes the manual step and records the result, the computer continues on with automated steps. This way if the majority of a test can be automated, a whole separate test does not need to be made just for the manual steps. In addition, the results from the manual step and automated steps are all in the same report generated by Toffee.

Manual tests are still a vital part of QA testing, and Toffee does not just accommodate manual testing, but it makes it less of a headache for testers and more accurate.

For more information about Toffee, visit http://toffeetesting.io, or feel free to contact us with questions at info@toffeetesting.io.

When working with clients and performing tests ourselves, we found that documenting tests with screenshot evidence is one of the most time consuming and challenging aspects of testing especially for manual tests. With this in mind, we designed Toffee to make evidence gathering easy with automatic screenshot capture for both manual and automated tests.

By using the simple command “enable automatic screenshot capture”, Toffee will begin to automatically take screenshots at every test step regardless if it is manual or automated. Testers no longer have to remember to do it themselves or repeat steps if they forget. The default timing for the screenshot is within milliseconds of the test step being completed, but the timing can be delayed as needed to accommodate slower applications. The screenshot delay can be modified at any point within the test. This allows flexibility to ensure the appropriate evidence is captured with every screenshot.

One of the main frustrations Toffee’s screenshot capture solves is keeping screenshots connected to the correct test step results. For example, without Toffee a tester often has to paste screenshots for manual tests into some additional word processing tool and document it appropriately so others know which screenshot corresponds to which test step. This method is tedious and prone to error. Toffee stores the results and screenshots together and can easily generate a single report containing all of the results and evidence in a single document.

Toffee makes gathering screenshot evidence easy by automatically capturing them after test steps and keeping them all in one place with the test results whether they are manual or automated.

For more information about Toffee, visit http://toffeetesting.io, or feel free to contact us with questions at info@toffeetesting.io.

Since my last post on functional testing, KSM has been hard at work transforming the ideas from that post into a new product and service offering. Toffee (“Test Orchestration for the Enterprise”) allows QA professionals to build and execute automated tests in an interactive, online test environment, without requiring programming expertise. For those test cases that automation cannot easily reach, Toffee lets you include manual test steps in your scripts alongside automated ones. Automated screenshots capture your entire desktop, providing evidence for steps whether executed within or outside of the browser. Test results for both automated and manual tests are presented in a familiar step/expected result/actual result format, along with screenshot evidence.

Toffee started as a command-line based solution, which allowed us to focus on the syntax and scope of the Toffee command set. We used this first incarnation to test solutions we developed in house. The largest test suite achieved 100% automation with over 28,000 test steps, and completely replaced the Selenium tests we had written in Java.

On the trade show floor of the Society of Quality Assurance Annual Meeting last week, KSM previewed the next generation of Toffee, called Toffee Composer. Composer provides the same level of functionality as the initial version, but in a user-friendly web interface. Build your scripts, execute them, and store your results online, either in our cloud-based environment or in your own data center.

KSM Technology Partners LLC is seeking solutions-focused professionals who enjoy using technology to solve complex, open-ended business problems. A KSM Consultant develops custom application and integration software for our clients in the pharmaceutical and utilities industries. The ideal candidate is a multi-disciplinary professional with 2+ years of experience writing great code in multiple languages, eliciting and documenting requirements, and designing and executing thorough unit and functional tests. Our consultants work in small, cross-functional, hyper-productive teams that deliver business value quickly and with a minimum of overhead.

Our offices and many of our clients are located in the greater Philadelphia area, particularly in the route 202 corridor in the western suburbs. Telecommuting is occasionally available for some engagements. We prefer local candidates.

This is a full-time position; we are not a body shop looking for subcontractors. We offer a competitive salary, flexible working hours, paid time off, healthcare and long-term disability coverage, and 401K with a generous match and no-wait vesting.

We will not consider candidates presented by agencies or other third parties. Applicants must be able to work unrestricted in the U.S.

If you are interested, email us at careers [/at/] ksmpartners [\dot\] com.

One of the key attributes I look for when writing and reviewing code is that code should express the intent of the developer more than the mechanism used to achieve that intent. In other words, code should read as much as possible as if it were a description of the end goal to be achieved. The mechanism used to achieve that goal is secondary.

Over the years, I’ve found this emphasis improves the quality of a system by making it easier to write correct code. By removing the distraction of the mechanism underneath the code: it’s easier for the author of that code to stay in the mindset of the business process they’re implementing. To see what I mean, consider how hard it would be to query a SQL database if every query was forced to specify the details of each table scan, index lookup, sort, join, and filter. The power of SQL is that it eliminates the mechanism of the query from consideration and lets a developer focus on the logic itself. The computer handles the details. Compilers do the same sort of thing for high level languages: coding in Java means not worrying about register allocation, machine instruction ordering, or the details of free memory reclamation. In the short-term, these abstractions make it easier to think about the problem I’m being paid to solve. Over a longer time scale, the increased distance between the intent and mechanism makes it easier to improve the performance or reliability of a system. Adding an index can transparently change a SQL query plan and Java seamlessly made the jump from an interpreter to a compiler.

One of the unique sources of power in the Lisp family of languages is a combination of features that makes it easier build the abstractions necessary to elevate code from mechanism to intent. The combination of dynamic typing, higher order functions, good data structures, and macros can make it possible to develop abstractions that allow developers to focus more on what matters, the intent of the paying customer, and less on what doesn’t. In this article, I’ll talk about what that looks like for the calculator example and how Clojure brings the tools needed to focus on the intent of the code.