related to timing the triggering of different subsys-tems. For example, an overlap-error scenario can
occur when the robot’s speed increases and switching between different paint spray patterns occurs too
rapidly. In principle, the control system should
detect overlap scenarios, send an appropriate error
message, and possibly shut down the system. However, it may happen that the robot does not work as
expected, for example due to bugs in the design or
implementation of the control system. Creating test
scenarios targeted at finding such subtle faults is nontrivial.

Faced with this challenge, researchers and engineers from Certus, a Norwegian research-based innovation center, have teamed up to explore the usage of
constraint programming (CP) for the validation of
ABB’s painting robots. Constraint-satisfaction techniques over finite domains were selected because of
their versatility in dealing with heterogeneous constraints derived from the design and implementation
of ABB’s control system for paint robots. We used a
combination of constraint propagation with different filtering consistencies and dedicated search
heuristics. With these techniques it was possible to
propose a cost-effective solution for test case generation for validating the control systems of ABB’s painting robots (Mossige, Gotlieb, and Meling 2015;
Mossige, Gotliev, and Meling 2014a, 2014b). Currently, these AI techniques are commonly supported
by modern CP solvers that ease their adoption in
industrial contexts. However, using a constraint
solver to generate tests can be time consuming,
which is at odds with the need to run both test generation and test execution as part of a continuous
integration cycle. Indeed, continuous integration is a
software engineering practice, where the result of test
execution, following a source code change, should be
reported back to the developer quickly.

Despite the strong technical culture of ABB Robotics and the availability of excellent textbooks,
deploying CP was not easy, as there are almost no
guidelines on how to model with constraints and
how to integrate CP in an industrial software production process. However, we managed to develop
such a model and to deploy it at ABB Robotics by
adding the constraint-solving process to the continuous integration process. The model has now been
part of ABB’s continuous integration process for more
than 24 months of daily operations. After having collected data about its bug-finding capabilities and efficiency, we perform a thorough analysis of its benefits
and drawbacks, which led us to draw some lessons
learned from this experience.

Bug-Finding Capabilitiesof ABB’s CP ModelFor the purpose of validating the CP model itself,known bugs were reintroduced in the paint controlsystems to examine the capabilities of the model tofind them. After some tuning, the model was able togenerate test scenarios that could find all the re-intro-duced bugs. This was considered a significant successand justified the continuation of the research andinnovation activities. Next, immediately after theintegration of the model in the software productionsystem, several unknown, but subtle bugs were foundrelated to the timing aspects of the paint control sys-tems. However, these bugs were classified as noncrit-ical, as they represented very unlikely scenarios.Upon further analysis, we observed that these bugshad been present in the control systems for severalyears without any significant consequences. Theywere corrected and the generated test scenarios wereadded to the testing process. However, although thebug-finding capabilities of the model had beenstrongly validated, ABB believed that its ability touncover mainstream bugs still remained to bedemonstrated.

Deploying CP in IndustrialSoftware ValidationThe validation of paint robots involves a fair amountof manual work, which is also error prone by nature.Therefore, deploying CP as a way to automate someparts of the process was welcomed by validation engi-neers and perceived as a way to strengthen the vali-dation process. However, CP also comes with somechallenges. The adoption of CP requires training, andthe maintenance of the CP model without expertassistance is difficult, especially since the automati-cally generated test scenarios are difficult to under-stand for untrained engineers. Tuning the model andits solving parameters, especially when optimizationprocedures are used, turned out to be reserved forexperts only. Validation engineers are usually skepti-cal about tools that produce results that are incom-prehensible to them, and almost impossible to com-pute by hand. To reduce the risks of rejection,internal training sessions on CP were organized andvarious front ends were built to help the engineersmanage the complexity of the generated test scenar-ios.

Return on Investment on Using CPTo compute the return on investment, one can meas-ure the number of uncovered bugs with and withoutthe CP model or compare the human effort requiredin both cases. However, for us, another more impor-tant factor was evaluating the possibly increased con-fidence of the engineers in the validation process —a factor almost impossible to quantify. After havingdeployed CP, we observed an increased appetiteamong engineers to refactor code covered by the CP-generated tests. Such refactoring is needed, but oftendeferred due to code complexity, which over time