3 How should we design a classifier for a given real world task?Abstraction of context, but may matter in a real-life application to a real-world problem.So improve by looking at details on particular instance to improve its usefulness for the particular problem.

4 Try Off-the-shelf ClassifiersMethod 1. No DesignTraining Setf(x)Test SetTry Off-the-shelf ClassifiersSVMLogistic-regressionDecision TreeNeural Network...Classification is a well-studied problem. There is no need of reinventing the wheel. We can just try a variety of standard classifiers to see which one works best.So far, we are very happy, we have a simple solution. Simplicity is beauty. What else do we need to do?We all forget to ask some simple questions: What’s the use of the classifier? Why do we care about a particular classification task at the first place? How do we measure the performance of a classifier, according to our interests?Most standard classifiers do answer what we care about. They simply assume we care about the error rate, and tries to minimize error rate, or equivalently, maximize accuracy.However, most standard classifiers are designed for minimizing error rate or maximizing accuracy. For many real world tasks, this is true. However, for some other real world tasks, minimizing error rate might not be exactly what we want to achieve in practice.Implicit Assumption: We are trying to minimize error rate, or equivalently, maximize accuracy

5 Method 2. Optimize what we really care aboutWhat’s the use of the classifier?How do we evaluate the performance of a classifier according to our interests?Quantify what we really care aboutOptimize what we care aboutCover very quickly here, and move this extended methodology to conclusion.Tightly couples performance evaluation and learning

8 We assume misclassification of products leads to revenue loss.Textual product description of a mouseProduct............Desktop computer and accessories...mousekeyboardpetrealize an expected annual revenuelose part of the potential revenue

9 What do we really care about?? maximize profitA vendor’s business goal is to maximize revenue, or equivalently, minimize revenue loss

10 Observation 1: the misclassification cost of a product depends on its potential revenue.

11 Desktop computer and accessoriesObservation 2: the misclassification cost of a product depends on how far apart the true class and the predicted class in the taxonomy.Textual product description of a mouseProduct.........Pet instead of car?...Desktop computer and accessories...mousekeyboardpet