I have developed an SVM-Model using x data. ROC curve was generated using 5-fold cross-validation. Now I want to compare my new SVM-model with a published Bayes-classifier. I have got the predictions for x data, using both published model and the SVM-model.

2 Answers
2

As Marc Claesen points out, some kind of certainty measure is needed. Below I have showed two approaches of how to form ROC curves.

If the classifier can output a probabilistic measure, such one can be used in e.g. 5-fold cross validation to form a ROC plot.

If the classifier only outputs predicted labels, then the certainty of predictions can be estimated with bagging. The training set is bootstrapped and modeled e.g. 100 times and the cross validated out-of-bag predictions are used for ROC curves.

Unfortunately, you can't generate ROC curves from that data. To construct ROC curves, you have to be able to rank the test set from strong positive to strong negative. For that you need scores, which measure the level of confidence that a data instance belong to the positive class (e.g. signed distance to the separating hyperplane for SVM and predicted probability for Naive Bayes).

$\begingroup$To construct an ROC curve you would need TPR and FPR pairs at each rank, not overall (overall is based on some arbitrary choice of threshold on the scores). The data must be the same for any reasonable comparison.$\endgroup$
– Marc ClaesenOct 8 '15 at 6:00

$\begingroup$Can you briefly explain the steps involved to draw the curve. It will be very helpful.$\endgroup$
– VAR121Oct 8 '15 at 17:10