Hi everyone, I am currently working with the mixOmics package. I was wondering whether I could add a 95% confidence interval to the AUC value. I am using different statistical models and want to compare them to each other. For the other models, I get an AUC + 95% CI, so I would like to get this as well for the sPLS-DA I am performing.
I used the auroc() function to get the ROC curve and the AUC value, but I am missing a 95% CI. I can’t think of a way to get to that 95% CI that works.
I know that I can create AUC + 95% CI with the pROC package. However, then I do not understand how to use my sPLS-DA model in the roc() function of this package.
Is there anybody that could help me out? It would help a lot with my master thesis!
Thanks in advance!
Kind regards,
Annemieke
hi @Annemieke,
It’s a technical question! You can access the R code here: mixOmics/R/auroc.R at master · mixOmicsTeam/mixOmics · GitHub
Roughly, it happens at line 121 where you get the prediction res.predict, and this is then use to get statauc.res which is the result of the plot.
So if you are R savvy you might be able to play with this and extract what you need 
(short answer is: I dont have a better quick fix for this)
Kim-Anh
Hi @kimanh.lecao ,
Thank you so much for your quick response!
I got it to work 
Hello! Same problem here. Could you explain how you did it? I am trying to extract the CI from resulting ROC curves from a tuned sPLS-DA model. Thanks! 
Hi @FGarrido!
I figured it out in a different way than described above. I tried to explain my code below! It is a bit of a long, weird way, but it works, so that is something I guess? And I checked whether it is correct with SPSS, so it should be alright.
I hope you understand the code below, let me know otherwise 
Good to know: My dataset is called training, my groups are defined within the column DiagnosisCategory (so diagnosis category is the “y” in my model)
final_sPLSDA <- splsda(x, y, ncomp = optimal_ncomp, keepX = optimal_nvariables) #final model
prediction <- predict(final_sPLSDA, x, dist = "mahalanobis.dist") #predict how well the model works
I made a table in which the discriminant scores and the diagnosis category are included with the following code
DiscriminantScoresTable <- cbind(as.data.frame(training$DiagnosisCategory) , as.data.frame(prediction$predict[,,"dim3"])) #the number in dimensions depends on the number of components included in your model
ROC <- roc(DiscriminantScoresTable$`data$DiagnosisCategory`, DiscriminantScoresTraining$NT1) #NT1 is one of my groups. The group you fill in is set as the case (I compared narcolepsy with healthy controls, so NT1 were my cases)
ROC$auc #for the AUC value
ci.auc(ROC$auc) #for the 95% confidence interval
2 Likes
Thanks for the reply!
That is a very intelligent solution! I was also playing with the predict function, but I thought that CI could be performed without the pROC package. Thanks a lot, it works perfectly
Just to note: inside the “roc” function, the NT1 column is selected from another data frame? I used my original DiscriminantScoreTable, which works just fine (it returns the same values as the auroc function).
Ah yeah, I see it! I tried to make my code a bit more understandable to post here by adjusting some names, but I forgot to adjust that
Glad it works!
1 Like