Linear Model Selection by Cross-Validation
Jun Shao · Journal of the American Statistical Association · 1993
We consider the problem of selecting a model having the best predictive ability among a class of linear models. The popular leave-one-out cross-validation method, which is asymptotically equivalent to many other model selection methods such as the Akaike information criterion (AIC), the C p , and the bootstrap, is asymptotically inconsistent in the sense that the probability of selecting the model with the best predictive ability does not converge to 1 as the total number of observations n → ∞. We show that the inconsistency of the leave-one-out cross-validation can be rectified by using a leave-n v -out cross-validation with n v , the number of observations reserved for validation, satisfying n v /n → 1 as n → ∞. This is a somewhat shocking discovery, because nv/n → 1 is totally opposite to the popular leave-one-out recipe in cross-validation. Motivations, justifications, and discussions of some practical aspects of the use of the leave-n v -out cross-validation method are provided, and results from a simulation study are presented.
Cited by 1,783 works. doi:10.1080/01621459.1993.10476299
Authors
References
- A new look at the statistical model identification
- Cross-Validatory Choice and Assessment of Statistical Predictions
- More Comments onCp
- A completely automatic french curve: fitting spline functions by cross validation
- Some Comments on C P
- Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation
- The Predictive Sample Reuse Method with Applications
- The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction
- How Biased is the Apparent Error Rate of a Prediction Rule?
- An Asymptotic Equivalence of Choice of Model by Cross-Validation and Akaike's Criterion