Tuesday, August 18, 2015

Manitest: Are classifiers really invariant? - implementation -

We recently published with Pascal Frossard a paper that introduces a
method to quantify the invariance of arbitrary classifiers to geometric
transformations. Our method is based on viewing the set of transformed
images as a smooth manifold and
define our invariance measure as a well-chosen geodesic distance on
that manifold. The paper is on the arXiv http://arxiv.org/abs/1507.06535 and the code is available on the project website https://sites.google.com/site/invmanitest/

Invariance to geometric transformations is a highly desirable property of
automatic classifiers in many image recognition tasks. Nevertheless, it is
unclear to which extent state-of-the-art classifiers are invariant to basic
transformations such as rotations and translations. This is mainly due to the
lack of general methods that properly measure such an invariance. In this
paper, we propose a rigorous and systematic approach for quantifying the
invariance to geometric transformations of any classifier. Our key idea is to
cast the problem of assessing a classifier's invariance as the computation of
geodesics along the manifold of transformed images. We propose the Manitest
method, built on the efficient Fast Marching algorithm to compute the
invariance of classifiers. Our new method quantifies in particular the
importance of data augmentation for learning invariance from data, and the
increased invariance of convolutional neural networks with depth. We foresee
that the proposed generic tool for measuring invariance to a large class of
geometric transformations and arbitrary classifiers will have many applications
for evaluating and comparing classifiers based on their invariance, and help
improving the invariance of existing classifiers.