Abstract

The Perceptron algorithm, despite its simplicity, often performs well on
online classification tasks. The Perceptron becomes especially effective
when it is used in conjunction with kernels. However, a common difficulty
encountered when implementing kernel-based online algorithms is the amount of
memory required to store the online hypothesis, which may grow
unboundedly. In this paper we present and analyze the Forgetron algorithm for
kernel-based online learning on a fixed memory budget. To our knowledge,
this is the first online learning algorithm which, on one hand, maintains a
{\em strict} limit on the number of examples it stores while, on the other
hand, entertains a relative mistake bound. In addition to the formal results,
we also present experiments with real datasets which underscore the merits of
our approach.