Keep having nearZerovar error

1

Hello,

It has been days that I try to fix this error that show p when I want to use the fonction per().

I have tried to use the fonction nearZerovar() or doing it manually. I verified after transformation if there were actually variables with zero variance left. But I always get this error. bellow is my script with the error warning:

perf.diablo = perf(basic.diablo.model, validation = ‘Mfold’,

  •                folds = 5, nrepeat = 5)

Error: There are features with zero variance in block ‘microbe’. If nearZeroVar() function or ‘near.zero.var’ parameter hasn’t been used, please use it. If you have used one of these, you may need to manually filter out these features.

I attach my table after nearZerovar() filtering

The filter I use is:
zero ← nearZeroVar(X2, freqCut = 12/1, uniqueCut = 15)

If you could help me to find the problem please?

After filtering

Column 1 Column 2 Column 3 Column 4 E F G H I J K L M N
0 2 0 0 0 0 0 0 0 0 0 0 0 4
0 0 2 0 0 0 0 0 0 0 0 0 0 4
0 0 0 0 0 0 0 0 3 0 0 4 0 0
0 5 0 0 0 0 0 0 2 0 0 0 0 0
0 4 0 0 0 0 0 0 0 0 0 0 0 4
0 5 0 0 0 0 0 0 0 0 0 0 0 4
0 0 5 0 0 0 0 0 4 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 4 0 0 5
0 0 0 0 0 0 0 0 0 6 0 2 0 0
0 0 4 0 0 0 0 0 0 0 0 0 0 7
0 0 0 7 0 0 0 0 4 0 0 0 0 0
0 0 7 0 0 0 0 0 0 0 0 0 0 4
0 8 0 0 0 0 0 0 0 2 0 0 0 0
0 5 0 0 0 0 0 0 0 0 0 0 0 7
0 8 0 0 0 0 0 0 0 0 0 0 0 3
0 6 0 0 0 0 0 0 0 0 0 0 0 7
0 9 0 0 0 0 0 0 0 0 0 0 0 2
0 0 0 0 0 0 0 0 9 0 0 3 0 0
0 0 0 0 6 4 0 0 0 7 0 0 0 0
0 4 0 0 0 0 0 0 0 0 0 0 0 9
0 0 4 0 0 0 0 0 0 0 10 0 0 0
0 4 0 0 0 0 0 0 0 0 0 0 0 10
0 2 0 0 0 0 0 0 0 0 11 0 0 0
0 0 11 0 0 0 0 0 0 2 2 0 0 0
0 8 9 0 0 0 0 0 0 0 0 0 0 5
0 0 11 0 0 0 0 0 0 0 0 0 0 6
0 0 9 0 0 0 0 0 0 0 0 0 0 9
0 0 0 0 0 0 0 4 0 12 0 0 0 0
0 4 0 0 0 0 0 0 0 0 0 0 0 13
0 10 8 0 0 0 0 0 0 0 0 0 0 8
0 0 0 0 0 0 0 0 2 0 10 0 0 11
0 4 14 0 0 0 0 0 0 0 0 0 0 0
0 10 11 0 0 0 0 0 0 0 0 0 0 0
0 12 9 0 0 0 0 0 0 0 0 0 0 0
0 11 11 0 0 0 0 0 2 0 2 0 0 0
0 15 0 0 0 0 0 0 0 0 0 0 0 2
0 7 0 0 0 0 0 0 0 0 0 0 0 14
0 0 15 0 0 0 0 0 0 0 0 0 0 6
0 9 12 0 0 0 0 0 0 0 0 0 0 9
0 10 0 0 0 0 0 0 0 0 0 0 0 15
0 0 0 0 0 0 2 6 0 0 8 16 0 0
0 0 13 0 0 0 0 0 0 0 0 0 0 13
0 0 13 0 0 0 0 0 0 0 0 0 0 13
0 4 9 0 0 0 0 0 0 0 0 0 0 16
0 7 17 0 0 0 0 0 5 0 0 4 0 8
0 0 0 0 0 0 0 0 0 2 14 14 0 0
0 0 11 0 0 0 0 0 0 0 0 0 0 17
0 5 0 0 0 0 0 0 0 0 8 0 0 19
0 5 19 0 0 0 0 0 0 0 0 0 0 10
0 0 0 0 0 0 0 0 0 0 6 0 0 21
0 5 3 0 0 0 0 0 0 0 0 0 0 23
0 0 5 0 0 0 0 0 0 0 0 0 0 23
0 4 5 0 0 0 0 0 0 0 13 9 0 20
0 11 18 0 0 0 0 0 0 0 0 0 0 14
0 4 18 0 0 0 0 0 0 0 0 0 0 17
0 0 24 0 0 0 0 0 0 0 3 6 0 0
0 0 14 0 0 0 0 0 0 0 0 0 0 21
0 17 19 0 0 0 0 0 0 0 0 0 0 0
0 0 25 0 0 0 0 0 3 0 3 0 0 0
0 0 25 0 0 0 0 0 0 0 0 0 0 5
0 0 25 0 0 0 0 0 0 0 0 0 0 6
0 5 2 0 0 11 4 3 5 9 2 0 0 25
0 3 23 0 0 0 0 0 0 0 0 0 0 15
0 18 22 0 0 0 0 0 0 0 0 0 0 4
0 7 12 0 0 0 0 0 0 0 0 0 0 26
0 0 13 0 0 0 0 0 0 0 0 0 0 26
0 22 21 0 0 0 0 0 0 0 0 0 0 7
0 0 0 0 0 0 0 0 0 0 0 6 0 30
0 0 32 0 0 0 0 0 0 0 0 0 0 5
0 0 29 0 0 0 0 0 0 0 0 0 0 21
0 0 34 0 0 0 0 0 4 0 0 4 0 11
0 0 14 0 0 0 0 0 0 0 0 0 0 33
0 21 27 0 0 0 0 0 0 0 0 0 0 16
0 9 35 0 0 0 0 0 0 0 0 0 0 0
0 0 35 0 0 0 0 0 0 0 0 0 0 10
0 22 0 0 0 0 0 0 0 0 0 0 0 30
0 9 28 0 0 0 0 0 0 0 0 0 0 26
0 0 0 0 0 0 0 37 0 0 5 0 0 0
0 23 27 0 0 0 0 0 0 0 0 0 0 19
0 0 35 0 0 0 0 0 0 0 0 0 0 23
0 3 42 0 0 0 0 0 0 0 0 0 0 7
0 0 39 0 0 0 0 0 0 0 5 0 0 21
0 12 8 0 0 0 0 0 0 0 0 0 0 42
0 30 30 0 0 0 0 0 0 0 0 0 0 21
0 7 42 0 0 0 0 0 0 0 0 0 0 17
0 8 41 0 0 0 0 0 0 0 0 0 0 20
0 0 44 0 0 0 0 0 0 0 0 0 0 12
0 12 44 0 0 0 0 0 4 0 0 0 0 22
0 43 0 0 0 26 0 0 0 0 0 0 0 0
0 0 48 0 0 0 0 0 0 0 0 0 0 14
0 0 0 0 0 0 48 0 0 0 15 0 0 0
0 7 49 0 0 0 0 0 0 0 0 0 0 12
0 0 0 0 0 2 0 0 50 0 0 0 0 0
0 26 33 0 0 0 0 0 0 0 0 0 0 38
0 3 31 0 0 0 0 0 0 0 0 0 0 46
0 0 35 0 0 0 0 0 6 0 0 5 0 44
0 37 51 0 0 0 0 0 0 0 0 0 0 20
0 0 0 8 0 0 40 0 0 0 26 0 0 48
0 17 59 0 0 0 0 0 0 0 0 0 0 24
0 0 54 0 0 0 0 0 0 0 0 0 0 38
0 7 51 0 0 0 0 0 40 0 19 5 0 28
0 30 40 0 0 0 0 0 0 0 0 0 0 49
0 17 63 0 0 0 0 0 0 0 0 0 0 24
0 49 47 0 0 0 0 0 0 0 0 0 0 29
0 7 69 0 0 0 0 0 0 0 0 0 0 14
0 30 41 0 0 0 0 0 0 0 0 2 0 55
0 0 65 0 0 0 0 0 0 0 0 0 0 40
0 33 33 0 0 0 0 0 0 0 0 0 0 64
0 11 50 0 0 0 0 0 0 0 0 0 0 61
0 26 62 0 0 5 0 0 4 0 0 0 0 50
0 23 78 0 0 0 0 0 0 0 0 0 0 26
0 12 86 0 0 0 0 0 0 0 0 6 0 24
0 0 0 0 0 0 0 0 0 48 0 82 0 0
0 14 88 0 0 0 0 0 0 0 0 0 0 35
0 27 0 56 0 0 0 0 0 0 0 2 0 79
0 6 90 0 0 0 0 0 0 0 0 0 0 41
0 0 0 0 0 0 0 0 87 53 0 0 0 0
0 19 72 0 0 0 0 18 0 0 0 0 0 75
0 18 96 0 0 0 0 0 0 0 0 0 0 39
0 27 47 0 0 0 0 0 0 0 0 0 0 99
0 6 0 0 0 0 3 0 0 108 0 0 0 0
0 6 104 0 0 0 0 0 0 0 0 0 0 42
0 0 0 0 0 0 0 0 0 111 30 0 0 0
0 22 87 0 0 0 0 0 3 0 0 0 0 84
0 0 118 0 0 0 0 0 0 0 0 0 0 17
0 0 0 0 0 0 0 123 0 0 2 0 0 0
0 23 112 0 0 0 0 0 0 0 0 0 0 66
0 2 0 0 0 0 0 0 0 127 0 5 0 23
0 45 71 0 0 0 0 0 0 2 0 0 0 124
0 13 131 0 0 0 0 0 0 0 0 0 0 97
0 47 94 0 0 0 0 0 0 3 0 0 0 141
0 23 162 0 0 0 0 0 0 0 0 0 0 49
0 26 149 0 0 0 0 0 0 0 0 0 0 86
0 6 82 161 0 0 0 0 0 0 0 0 0 10
0 26 160 0 0 0 0 0 0 0 0 0 0 101
0 45 166 0 2 0 0 0 0 0 0 0 0 97
0 16 161 0 0 0 0 0 0 0 0 0 0 120
0 29 181 0 0 0 0 0 14 5 12 9 0 120
0 0 0 79 0 0 0 0 0 0 0 195 0 0
0 17 157 0 0 0 0 0 0 0 0 0 0 146
0 17 208 0 0 0 0 0 0 0 0 0 0 29
0 36 113 192 0 0 0 0 0 0 0 0 0 24
0 0 0 83 0 0 0 0 203 69 0 12 0 0
0 65 214 0 0 8 0 0 0 2 0 0 0 72
0 0 21 234 0 0 0 0 0 0 0 0 0 18
0 17 238 0 0 0 0 0 0 0 0 0 0 64
0 0 177 190 0 0 0 0 0 0 0 0 0 0
0 0 0 281 0 0 0 0 0 0 0 0 0 14
0 0 17 0 0 0 0 0 0 0 0 283 0 0
0 0 289 0 0 0 0 0 0 0 7 0 0 0
0 0 0 287 2 0 0 0 0 7 75 0 0 0
0 0 0 0 0 40 0 293 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 336 0 4
0 3 20 199 0 0 0 0 0 305 0 0 0 19
0 0 9 236 0 0 0 0 0 280 0 0 0 25
0 3 0 0 159 324 0 0 0 0 0 0 0 0
0 0 0 100 0 347 0 0 0 0 0 4 0 0
0 37 357 0 0 0 0 0 0 0 0 0 0 121
0 52 356 0 0 0 0 0 0 0 0 0 0 126
0 0 0 195 0 320 0 0 163 0 0 0 0 0
0 0 0 0 0 390 0 0 2 0 0 0 0 0
0 10 0 0 0 0 0 396 0 0 0 0 0 0
0 0 0 0 0 0 0 425 0 0 0 3 0 3
0 12 0 162 58 82 0 156 0 0 0 416 0 0
0 66 386 0 0 0 0 0 0 0 0 0 0 254
0 0 4 0 0 0 0 463 0 0 0 0 0 0
0 0 0 468 0 0 0 41 0 0 0 0 0 0
0 80 318 0 0 0 0 0 0 0 0 7 0 385
0 0 0 151 0 0 0 397 0 353 0 2 0 0
0 88 444 0 0 0 0 0 0 0 0 0 0 326
0 0 531 11 0 0 29 0 25 12 9 131 0 0
0 382 37 254 0 333 0 287 0 0 3 0 0 12
0 162 480 0 0 0 0 0 0 0 0 0 0 324
0 0 16 0 0 0 0 0 0 0 0 599 0 14
0 85 0 24 0 0 624 0 0 0 16 17 0 0
0 0 0 0 0 399 0 0 569 79 0 0 0 0
0 0 0 0 0 70 0 58 137 0 0 681 0 0
0 148 629 0 0 0 0 0 18 0 4 7 0 316
0 0 0 25 293 0 0 347 0 0 374 536 0 2
0 359 0 83 0 103 0 10 0 33 666 211 0 4
0 0 0 472 0 431 305 201 0 0 404 318 0 0
0 0 0 0 0 603 0 347 133 10 417 0 0 0
0 0 0 67 0 535 570 158 0 0 0 0 0 4
0 0 0 0 0 623 0 0 563 0 0 0 0 0
0 763 0 203 51 14 0 0 0 408 0 0 0 0
0 0 0 0 0 0 0 0 848 79 0 0 0 0
0 142 547 5 0 18 167 0 0 19 10 7 0 717
0 135 800 0 0 0 0 0 2 0 10 0 0 376
0 476 0 639 84 260 506 307 0 0 0 0 0 0
0 0 0 719 0 0 0 0 4 553 0 0 0 0
0 381 909 40 0 297 164 144 4 134 207 3 0 0
0 0 0 0 0 0 0 0 479 0 836 0 0 0
0 0 0 0 0 0 0 402 888 0 0 0 0 0
0 0 0 880 0 0 532 211 183 157 0 0 0 0
0 973 0 0 0 0 257 0 0 0 0 0 0 0
0 704 369 4 0 2 587 62 85 4 2 621 0 114
0 0 0 0 838 0 0 0 813 0 0 0 0 0
0 515 0 914 0 0 0 0 655 0 0 0 0 0
0 116 342 188 830 0 436 369 705 524 952 0 0 136
0 902 33 105 0 0 0 0 0 0 0 0 0 997
0 589 0 0 0 0 0 185 0 0 0 0 0 1564
0 412 3 248 0 273 560 310 635 169 650 860 0 1579
0 0 0 266 0 0 1357 1046 0 0 0 0 0 0
0 3 31 460 119 0 482 0 1690 0 0 0 0 31
0 0 126 0 0 0 0 24 0 0 1114 1650 0 0
0 524 1909 135 292 70 367 131 949 164 906 626 0 798
0 0 0 1749 0 0 1111 490 226 384 0 0 0 0
2 494 1848 0 0 0 0 0 2 2 0 4 0 970
0 0 17 0 0 0 0 0 0 0 0 0 0 2059
0 0 2105 0 0 0 0 0 0 0 0 0 0 818
0 0 2218 516 0 0 0 0 0 0 0 0 0 0
0 3 2684 0 0 0 0 0 0 226 0 0 0 0
0 68 264 737 0 334 2478 265 1806 558 3 3 0 124
0 151 0 337 2802 786 3 937 0 0 337 596 0 0
0 69 340 814 0 828 2652 216 3128 1556 964 0 0 207
0 641 3514 0 0 3 0 0 0 0 0 0 0 1927
0 42 4326 0 0 0 0 0 4 0 0 0 0 0
0 0 3097 2436 2428 2803 4077 2465 1104 736 1778 499 0 2401
0 0 0 2684 3678 1290 1653 2522 3321 0 192 0 0 0
0 5 0 0 0 0 0 0 0 0 0 5395 0 0
0 6 7 1145 7282 3060 244 278 1023 0 0 243 0 11
0 1898 8463 1356 2411 449 2021 977 1825 735 1662 549 0 23
0 0 0 5438 4898 2899 6780 4470 5164 655 7559 1844 0 382
0 1002 8035 2123 10316 1881 1680 1639 2597 3123 6166 3960 0 0
0 4866 3090 8832 5847 4729 8033 7299 5881 6775 12622 6181 3 2789
0 5 2 5873 8406 4746 6277 7147 8982 872 4674 4181 0 0
5 3854 10975 3959 1798 2559 1463 8180 270 3161 9431 5935 0 7320
6 7294 35974 6807 18595 10429 5298 6067 10354 5975 13834 14981 0 20437
0 3901 4070 33243 23388 22412 25766 29811 38085 4284 34669 11024 0 5222
11 4549 6659 92111 87448 96893 72972 77810 80157 18783 77234 25669 0 17888
0 1732 54352 8793 0 24594 27816 29206 20946 135493 3607 41359 11703 93430
187887 142651 0 3635 5189 5659 1263 750 1249 175 2881 3007 171628 9
2

hi @Geraldinevit

The problem is that you still have too many zeroes, especially during cross-validation where we consider only 4/5 of the rows (it is pretty obvious from your data).
Here are some solutions:

  • filter the dataset more drastically by using the default arguments, i.e zero ← nearZeroVar(X2)
  • try a different M fold value in cross validation, i.e folds = 3, nrepeat = 10 (in general we advise a larger value nrepeat = 50)

Or a combination of the two above.

Kim-Anh