Kim Herzig

Predicting Defects in SAP Products: A Replicated Study

Given a large body of code, how do we know where to focus our quality assurance effort? By mining the software’s defect history, we can automatically learn which code features correlated with defects in the past—and leverage these correlations for new predictions: “In the past, high inheritance depth was an indicator of a high number of defects. Since this new component also has a high inheritance depth, let us test it thoroughly”. Such history-based approaches work best if the new component is similar to the components learned from. But how does learning from history perform for projects with high variability between components? We ran a study on two SAP products involving a wide spectrum of functionality. We found that learning and predicting was accurate at package level, but not at product level. These results suggest that to learn from past defects, one should separate the product into component clusters with similar functionality, and make separate predictions for each cluster. Our initial approaches to form such clusters automatically, based on similarity of metrics, showed promising accuracy.