Statistical tests using hinge/ε-sensitive loss
Citation
Yıldız, O. T. & Alpaydın, A. İ. E. (2013). Statistical tests using hinge/ε-sensitive loss. Paper presented at the Computer and Information Sciences III - 27th International Symposium on Computer and Information Sciences, ISCIS 2012, 153-160. doi:10.1007/978-1-4471-4594-3-16Abstract
Statistical tests used in the literature to compare algorithms use the misclassification error which is based on the 0/1 loss and square loss for regression. Kernel-based, support vector machine classifiers (regressors) however are trained to minimize the hinge (ε-sensitive) loss and hence they should not be assessed or compared in terms of the 0/1 (square loss) but with the loss measure they are trained to minimize. We discuss how the paired t test can use the hinge (ε-sensitive) loss and show in our experiments that doing that, we can detect differences that the test on error cannot detect, indicating higher power in distinguishing between the behavior of kernel-based classifiers (regressors). Such tests can be generalized to compare L > 2 algorithms.