Fascinating angle at the quality of ML models (if you define quality as “can I break this?”). The adversarial approach is akin to “bad inputs” testing of regular software, only the test data is trickier to craft, and the results are much more far-reaching: a continuously learning model will produce wrong outputs for many other inputs once the “bad” data has been fed in.

DeepXplore testing method relies on having at least 3 neural networks for the same problem space so their solutions can be cross-checked against each other. The approach aims to maximize “neuron coverage” (similar to code coverage, a measure of network pathways that get activated during the test). It appears to be quite effective in finding bugs.

If your company has a data science team, chances are, they can use an introduction to testing. Do not expect that DS folks understand the value of unit testing, have heard of TDD, and are armed with debugging skills. Quality engineers can make a tremendous difference by providing training and support to DS in this area. These posts will come in handy.

Off-Topic

If you received this email directly then you’re already signed up, thanks! Else
if this newsletter issue was forwarded to you and you’d like to get one weekly,
then you can subscribe at http://testersdigest.mehras.net