Hi everyone, I am currently working with the mixOmics package. I was wondering whether I could add a 95% confidence interval to the AUC value. I am using different statistical models and want to compare them to each other. For the other models, I get an AUC + 95% CI, so I would like to get this as well for the sPLS-DA I am performing.
I used the auroc() function to get the ROC curve and the AUC value, but I am missing a 95% CI. I can’t think of a way to get to that 95% CI that works.
I know that I can create AUC + 95% CI with the pROC package. However, then I do not understand how to use my sPLS-DA model in the roc() function of this package.
Is there anybody that could help me out? It would help a lot with my master thesis!
Thanks in advance!
Kind regards,
Annemieke
hi @Annemieke,
It’s a technical question! You can access the R code here: mixOmics/R/auroc.R at master · mixOmicsTeam/mixOmics · GitHub
Roughly, it happens at line 121 where you get the prediction res.predict
, and this is then use to get statauc.res
which is the result of the plot.
So if you are R savvy you might be able to play with this and extract what you need ![:grimacing: :grimacing:](https://emoji.discourse-cdn.com/apple/grimacing.png?v=12)
(short answer is: I dont have a better quick fix for this)
Kim-Anh
Hi @kimanh.lecao ,
Thank you so much for your quick response!
I got it to work ![:tada: :tada:](https://emoji.discourse-cdn.com/apple/tada.png?v=12)
Hello! Same problem here. Could you explain how you did it? I am trying to extract the CI from resulting ROC curves from a tuned sPLS-DA model. Thanks! ![:slight_smile: :slight_smile:](https://emoji.discourse-cdn.com/apple/slight_smile.png?v=12)
Hi @FGarrido!
I figured it out in a different way than described above. I tried to explain my code below! It is a bit of a long, weird way, but it works, so that is something I guess? And I checked whether it is correct with SPSS, so it should be alright.
I hope you understand the code below, let me know otherwise ![:slightly_smiling_face: :slightly_smiling_face:](https://emoji.discourse-cdn.com/apple/slightly_smiling_face.png?v=12)
Good to know: My dataset is called training, my groups are defined within the column DiagnosisCategory (so diagnosis category is the “y” in my model)
final_sPLSDA <- splsda(x, y, ncomp = optimal_ncomp, keepX = optimal_nvariables) #final model
prediction <- predict(final_sPLSDA, x, dist = "mahalanobis.dist") #predict how well the model works
I made a table in which the discriminant scores and the diagnosis category are included with the following code
DiscriminantScoresTable <- cbind(as.data.frame(training$DiagnosisCategory) , as.data.frame(prediction$predict[,,"dim3"])) #the number in dimensions depends on the number of components included in your model
ROC <- roc(DiscriminantScoresTable$`data$DiagnosisCategory`, DiscriminantScoresTraining$NT1) #NT1 is one of my groups. The group you fill in is set as the case (I compared narcolepsy with healthy controls, so NT1 were my cases)
ROC$auc #for the AUC value
ci.auc(ROC$auc) #for the 95% confidence interval
2 Likes
Thanks for the reply!
That is a very intelligent solution! I was also playing with the predict function, but I thought that CI could be performed without the pROC package. Thanks a lot, it works perfectly ![:slight_smile: :slight_smile:](https://emoji.discourse-cdn.com/apple/slight_smile.png?v=12)
Just to note: inside the “roc” function, the NT1 column is selected from another data frame? I used my original DiscriminantScoreTable, which works just fine (it returns the same values as the auroc function).
Ah yeah, I see it! I tried to make my code a bit more understandable to post here by adjusting some names, but I forgot to adjust that ![:slight_smile: :slight_smile:](https://emoji.discourse-cdn.com/apple/slight_smile.png?v=12)
Glad it works!
1 Like