Dear AFNI gurus,

I'm trying to understand the permutation tests provided by 3dttest++, especially the ETAC method which is truly an ingenious solution to the cluster defining p trade-off problem. I have a question though, regarding the construction of the permuted, pure-noise dataset.
In the simplest permutation tests (comparing two-sample or one-sample mean), we permute or sign-flip the raw observations under the null hypothesis that there is no effect.
For more complex GLMs with covariates, one can permute the residuals of the reduced model (no effects of interest, the observations are explained only by the nuisance effects), as suggested in (Winkler et al., 2014).
However, the algorithm behind -ETAC and -Clustsim seems to permute the residuals of the full GLM model, according to the 2017 and 2019 paper. (I'm not sure whether I understood this part correctly.) This means, the permuted datasets will not be able to recreate the original observed data even in theory.
Would this somehow cause the reference distribution to contain fewer "extreme" samples and become narrower than it should be, and lead to an inflated p-value?

I did a simulation for the permutation test of one-sample mean using zero-mean Gaussian noise data. If I sign-flipped the raw data, the resulting p-values were uniformly distributed in [0,1] as expected. However, if I removed the mean of the sample (not zero due to random fluctuation) first, and permuted the residuals, the p distribution showed a peak below p=0.05. These results seem to support the above concern.

Could you make it more explicit about how the residuals are generated for the permutation procedure?

Best regards,
Chencan



Edited 2 time(s). Last edit at 01/16/2023 11:29AM by herrlich10.