Hi @psd,
From my understanding, a permutation test should:
- Create a ‘final’ model (in your case the sPLS model)
- Apply the loadings from the above model to test datasets where Y is shuffled
- Look at the distribution of error rates/component correlations
This should give a unimodal distribution, not a bimodal one. Judging from your text I’m wondering whether you have been applying new loadings from your permuted test dataset (made in step 2) to the test data (also made in step 2) rather than the loadings from your original model (made in step 1) to your permuted test data (made in step 2). You can have a look at this related post to see some example code of a permutation test and compare to your own.
Hope that helps,
Eva