A permutation test sensitive to differences in areas for comparing ROC curves from a paired design
Andriy I. Bandos, Howard E. Rockette, David Gur · Statistics in Medicine · 2005
The area under the receiver operating characteristic (ROC) curve (AUC) is a widely accepted summary index of the overall performance of diagnostic procedures and the difference between AUCs is often used when comparing two diagnostic systems. We developed an exact non-parametric statistical procedure for comparing two ROC curves in paired design settings. The test which is based on all permutations of the subject specific rank ratings is formally a test for equality of ROC curves that is sensitive to the alternatives of AUC difference. The operating characteristics of the proposed test were evaluated using extensive simulations over a wide range of parameters. The proposed procedure can be easily implemented in experimental ROC data sets. For small samples and for underlying parameters that are common in experimental studies in diagnostic imaging the test possesses good operating characteristics and is more powerful than the conventional non-parametric procedure for AUC comparisons. We also derived an asymptotic version of the test which uses an exact estimate of the variance in the permutation space and provides a good approximation even when the sample sizes are small. This asymptotic procedure is a simple and precise approximation to the exact test and is useful for large sample sizes where the exact test may be computationally burdensome.
Cited by 68 works. doi:10.1002/sim.2149
Authors
References
- The meaning and use of the area under a receiver operating characteristic (ROC) curve.
- Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach
- Basic principles of ROC analysis
- A method of comparing the areas under receiver operating characteristic curves derived from the same cases.
- A Class of Statistics with Asymptotically Normal Distribution
- Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals—Rating-method data
- The area above the ordinal dominance graph and the area below the receiver operating characteristic graph
- Evaluation of diagnostic systems : methods from signal detection theory
- A Class of Statistics with Asymptotically Normal Distribution
- ROC Methodology in Radiologic Imaging