HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Full Text Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Cherkassky, V. Articles by Ma, Y. Search for Related Content PubMed PubMed Citation Articles by Cherkassky, V. Articles by Ma, Y.

Comparison of Model Selection for Regression

cherkass{at}ece.umn.edu Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, MN 55455, U.S.A.

Yunqian Ma

myq{at}ece.umn.edu Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, MN 55455, U.S.A.

We discuss empirical comparison of analytical methods for model selection. Currently, there is no consensus on the best method for finite-sample estimation problems, even for the simple case of linear estimators. This article presents empirical comparisons between classical statistical methods—Akaike information criterion (AIC) and Bayesian information criterion (BIC)—and the structural risk minimization (SRM) method, based on Vapnik-Chervonenkis (VC) theory, for regression problems. Our study is motivated by empirical comparisons in Hastie, Tibshirani, and Friedman (2001), which claims that the SRM method performs poorly for model selection and suggests that AIC yields superior predictive performance. Hence, we present empirical comparisons for various data sets and different types of estimators (linear, subset selection, and k-nearest neighbor regression). Our results demonstrate the practical advantages of VC-based model selection; it consistently outperforms AIC for all data sets. In our study, SRM and BIC methods show similar predictive performance. This discrepancy (between empirical results obtained using the same data) is caused by methodological drawbacks in Hastie et al. (2001), especially in their loose interpretation and application of SRM method. Hence, we discuss methodological issues important for meaningful comparisons and practical application of SRM method. We also point out the importance of accurate estimation of model complexity (VC-dimension) for empirical comparisons and propose a new practical estimate of model complexity for k-nearest neighbors regression.

This article has been cited by other articles:

				D. L. Dowe, S. Gardner, and G. Oppy Bayes not Bust! Why Simplicity is no Problem for Bayesians Brit J Philos Sci, December 1, 2007; 58(4): 709 - 754. [Abstract] [Full Text] [PDF]

				T. Hastie, R. Tibshirani, and J. Friedman Note on "Comparison of Model Selection for Regression" by Vladimir Cherkassky and Yunqian Ma Neural Comput., July 1, 2003; 15(7): 1477 - 1480. [Abstract] [Full Text] [PDF]