Robustness and Regularization of Support Vector Machines

Abstract

We consider regularized support vector machines (SVMs) and show that
they are precisely equivalent to a new robust optimization
formulation. We show that this equivalence of robust optimization
and regularization has implications for both algorithms, and
analysis. In terms of algorithms, the equivalence suggests more
general SVM-like algorithms for classification that explicitly build
in protection to noise, and at the same time control overfitting. On
the analysis front, the equivalence of robustness and regularization
provides a robust optimization interpretation for the success of
regularized SVMs. We use this new robustness interpretation of SVMs
to give a new proof of consistency of (kernelized) SVMs, thus
establishing robustness as the reason regularized SVMs
generalize well.