Dear Kim-Anh,

many thanks for your additional reply and particularly the notes and references you have provided. I will look more into these. Thanks again!

I did think about permutation tests, in the way that you will permute the labels/ end-point and check if you still obtained similar separation or not after you have changed the structure, indicating if the original was a true relation or spurious signals. However, in a two-class problem, swapping the labels randomly will still leave some with the same ids as before and one will also need a check indicating the overap with original data labels. Thus, the permutation test will have to be displayed on two axis for a series of permutations, so to speak - overlap fraction of permuted labels with original & thus obtained signal value. Lastly, swapping all labels with another in a two class problem just calls A as B, and B as A but maintains inherently the same data structure.

Is my understanding correct in the way you alluded to using the permutation tests, and the limitation I highlight for binary classification problems?

Thank you again in advance for your insights and valuable discussions!

SKD