In many areas of AI decades of research have resulted in many different approaches to solving similar problems. These different approaches exhibit different performance
characteristics on different problem types, and it is usually
unclear when to choose which approach. This is known as
the algorithm selection problem (Rice 1976).

The challenge of algorithm selection in practice has led to
the development of various data-driven, automated solutions
to it. Machine-learning techniques are used to transparently
select the most appropriate algorithm for the problem at
hand (O’Mahony et al. 2008, Hurley et al. 2014, Xu et al.
2008). Such systems have demonstrated that significant performance improvements can be achieved over using just a
single approach. The interested reader is referred to a recent
survey for more information (Kotthoff 2014).

n Algorithm selection is of increasing
practical relevance in a variety of applications. Many approaches have been
proposed in the literature, but their evaluations are often not comparable, making it hard to judge which approaches
work best. The ICON Challenge on
Algorithm Selection objectively evaluated many prominent approaches from
the literature, making them directly
comparable for the frst time. The
results show that there is still room for
improvement, even for the very best
approaches.