False alarms are a big challenge for Intrusion Detection Systems (IDSs). A lot of approaches, especially machine learning based schemes, have been proposed to mitigate this issue by filtering out these false alarms. But a fundamental problem is how to objectively evaluate an algorithm in terms of its ability to correctly identify false alarms and true alarms. To improve the utilization of various machine learning algorithms, intelligent false alarm reduction has been proposed that aims to select and apply an appropriate algorithm in an adaptive way. Traditional metrics (e.g., true positive rate, false positive rate) are mainly used in the algorithm selection and evaluation, however, no single metric seems sufficient and objective enough to measure the capability of an algorithm in reducing false alarms.