The Geometry of Losses

Abstract

Loss functions are central to machine learning because they are the means by which the quality of a prediction is evaluated. Any loss that is not proper, or can not be transformed to be proper via a link function is inadmissible. All admissible losses for n-class problems can be obtained in terms of a convex body in \mathbbR^n. We show this explicitly and show how some existing results simplify when viewed from this perspective. This allows the development of a rich algebra of losses induced by binary operations on convex bodies (that return a convex body). Furthermore it allows us to define an “inverse loss” which provides a universal “substitution function” for the Aggregating Algorithm. In doing so we show a formal connection between proper losses and norms.

Related Material

@InProceedings{pmlr-v35-williamson14,
title = {The Geometry of Losses},
author = {Robert C. Williamson},
booktitle = {Proceedings of The 27th Conference on Learning Theory},
pages = {1078--1108},
year = {2014},
editor = {Maria Florina Balcan and Vitaly Feldman and Csaba Szepesvári},
volume = {35},
series = {Proceedings of Machine Learning Research},
address = {Barcelona, Spain},
month = {13--15 Jun},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v35/williamson14.pdf},
url = {http://proceedings.mlr.press/v35/williamson14.html},
abstract = {Loss functions are central to machine learning because they are the means by which the quality of a prediction is evaluated. Any loss that is not proper, or can not be transformed to be proper via a link function is inadmissible. All admissible losses for n-class problems can be obtained in terms of a convex body in \mathbbR^n. We show this explicitly and show how some existing results simplify when viewed from this perspective. This allows the development of a rich algebra of losses induced by binary operations on convex bodies (that return a convex body). Furthermore it allows us to define an “inverse loss” which provides a universal “substitution function” for the Aggregating Algorithm. In doing so we show a formal connection between proper losses and norms. }
}

%0 Conference Paper
%T The Geometry of Losses
%A Robert C. Williamson
%B Proceedings of The 27th Conference on Learning Theory
%C Proceedings of Machine Learning Research
%D 2014
%E Maria Florina Balcan
%E Vitaly Feldman
%E Csaba Szepesvári
%F pmlr-v35-williamson14
%I PMLR
%J Proceedings of Machine Learning Research
%P 1078--1108
%U http://proceedings.mlr.press
%V 35
%W PMLR
%X Loss functions are central to machine learning because they are the means by which the quality of a prediction is evaluated. Any loss that is not proper, or can not be transformed to be proper via a link function is inadmissible. All admissible losses for n-class problems can be obtained in terms of a convex body in \mathbbR^n. We show this explicitly and show how some existing results simplify when viewed from this perspective. This allows the development of a rich algebra of losses induced by binary operations on convex bodies (that return a convex body). Furthermore it allows us to define an “inverse loss” which provides a universal “substitution function” for the Aggregating Algorithm. In doing so we show a formal connection between proper losses and norms.

TY - CPAPER
TI - The Geometry of Losses
AU - Robert C. Williamson
BT - Proceedings of The 27th Conference on Learning Theory
PY - 2014/05/29
DA - 2014/05/29
ED - Maria Florina Balcan
ED - Vitaly Feldman
ED - Csaba Szepesvári
ID - pmlr-v35-williamson14
PB - PMLR
SP - 1078
DP - PMLR
EP - 1108
L1 - http://proceedings.mlr.press/v35/williamson14.pdf
UR - http://proceedings.mlr.press/v35/williamson14.html
AB - Loss functions are central to machine learning because they are the means by which the quality of a prediction is evaluated. Any loss that is not proper, or can not be transformed to be proper via a link function is inadmissible. All admissible losses for n-class problems can be obtained in terms of a convex body in \mathbbR^n. We show this explicitly and show how some existing results simplify when viewed from this perspective. This allows the development of a rich algebra of losses induced by binary operations on convex bodies (that return a convex body). Furthermore it allows us to define an “inverse loss” which provides a universal “substitution function” for the Aggregating Algorithm. In doing so we show a formal connection between proper losses and norms.
ER -

Williamson, R.C.. (2014). The Geometry of Losses. Proceedings of The 27th Conference on Learning Theory, in PMLR 35:1078-1108