PLS and Permutation Testing

1

Greetings,

I am hoping to use sparse PLS in MixOmics in a train/test context, where I first build a sPLS model with my training dataset (while using cross-validation with this training dataset to find the optimal hyperparameters of number of components and sparsity levels for my X and Y matrices).

I then want to apply the final trained sPLS model to a test dataset. To do so, I take the loading vectors from the trained model and apply them to the X and Y matrices of my test data. I then find the correlation between the first component in my test data and want to see if it’s significant through permutation testing.

To do so, for each permutation loop, I simply shuffle the Y data from the trained data, create a sPLS model with the now X and shuffled Y training dataset, apply the loadings to my test dataset, and recalculate the correlation between the first component. I want to see where my “real” test correlation falls in this distribution of correlations obtained with shuffled data.

My concern is from the fact that when I look at my histogram of permutated correlation values, there seems to be a clear bimodal nature to this distribution:

image
image1214×1008 32.3 KB

I want to ask if 1) this is to be expected with permutation approaches with PLS when applying a trained model to a test dataset, and 2) if not, what could I be doing wrong?

Thank you in advance.

2

Hi @psd,

From my understanding, a permutation test should:

  1. Create a ‘final’ model (in your case the sPLS model)
  2. Apply the loadings from the above model to test datasets where Y is shuffled
  3. Look at the distribution of error rates/component correlations

This should give a unimodal distribution, not a bimodal one. Judging from your text I’m wondering whether you have been applying new loadings from your permuted test dataset (made in step 2) to the test data (also made in step 2) rather than the loadings from your original model (made in step 1) to your permuted test data (made in step 2). You can have a look at this related post to see some example code of a permutation test and compare to your own.

Hope that helps,
Eva