Statistical learning and selective inference
Jonathan Taylor, Robert John Tibshirani · Proceedings of the National Academy of Sciences · 2015
We describe the problem of "selective inference." This addresses the following challenge: Having mined a set of data to find potential associations, how do we properly assess the strength of these associations? The fact that we have "cherry-picked"--searched for the strongest associations--means that we must set a higher bar for declaring significant the associations that we see. This challenge becomes more important in the era of big data and complex statistical modeling. The cherry tree (dataset) can be very large and the tools for cherry picking (statistical learning methods) are now very sophisticated. We describe some recent new developments in selective inference and illustrate their use in forward stepwise regression, the lasso, and principal components analysis.
Cited by 395 works. doi:10.1073/pnas.1507583112
Authors
References
- Why Most Published Research Findings Are False
- Statistical significance for genomewide studies
- Why Most Published Research Findings Are False
- The Netflix Prize
- For most large underdetermined systems of linear equations the minimal 𝓁 1 ‐norm solution is also the sparsest solution
- Statistical Learning with Sparsity
- Human Immunodeficiency Virus Reverse Transcriptase and Protease Sequence Database
- KDD Cup and workshop 2007
- Human immunodeficiency virus reverse transcriptase and protease sequence database
- Valid post-selection inference