Announcing the Dissertation Defense of Mr. Ngoc Ty Nguyen for the degree of Doctor of Big Data Analytics
Abstract: The receiver operating characteristic (ROC) curve is used to measure classification accuracy of tests which yield ordinal or continuous scores. Ordinal scores occur commonly in medical imaging studies and more recently in black-box studies on forensic identification accuracy (Phillips et al.,2018). To assess the accuracy of radiologists in medical imaging studies or the accuracy of forensic examiners in biometric studies, one needs to estimate the ROC curves from the ordinal scores and also account for the covariates related to the radiologists or forensic examiners. In this thesis, we propose a homogeneity test to compare performance of raters. Asymptotic properties of estimated ROC curves and their corresponding AUCs within ordinal regression framework are derived as well. Moreover, differences in ROC curves (and in AUCs) among examiners are also investigated in detail. Confidence intervals of difference in AUCs and confidence bands of difference in ROC curves are built up for performance comparison purposes. First, simulations are conducted on data where scores are assumed to be normal, and features are both categorical and continuous covariates. Then, we apply our procedure to facial recognition data to compare forensic examiners.
The second part of this thesis considers correlation of decision scores among raters. In medical imaging studies and in facial recognition, because multiple raters assess the same subject pairs, the scores from the readers may be correlated. Due to the correlated scores, standard methods for generalized linear models cannot be applied directly to estimate accuracy. Moreover, characteristics of subjects or examiners such as age, gender, race may affect the accuracy of tests. Thus, those features should be adjusted when the accuracy of raters are estimated. In this thesis, we estimate covariate-specific and covariate-adjusted AUCs using the generalized estimating equation for ordinal scores when correlations are involved. Homogeneity tests on covariate-specific and covariate-adjusted AUCs will be conducted and the statistical properties of the tests will be investigated. Simulation studies will be conducted to evaluate the finite sample properties of the homogeneity test. The test will be applied to real facial recognition data.
Committee in Charge:
Dr. Larry Tang (Chair)
Dr. Rui Xie
Dr. Edgard Maboudou
Dr. Michael Sigman