> You mean that doing such a comparison could be meaningless or at least is hard to interpret?

I was just concerned about the interpretation. But if you don't think that is an issue, it should be fine.

> To get a firmer grasp on this I played a bit with the effect of scaling when adding effects
> of different sizes to the data, where two sets of data points are generated, one much
> stronger than the other
>
> The four plots show sorted p-values for four different strengths of effects (0, 1, 2, and 3; arbitrary
> units) using 1,000 iterations per plot. Without normalization all p-values are extremely significant
> (except when there is no effect). With sos normalization they behave a lot better, though a little
> bit on the conservative side.
>
> Do you think this is a valid simulation and would that mean that the sos normalization would be
> sufficiently valid (although a bit conservative)?

The paired t-test is a special one-way repeated-measures ANOVA in the sense that there are only two levels in the repeated-measures factor. Such a model has a sphericity assumption, which means in this case that the variance is assumed same between the two maps. Apparently the assumption is significantly violated because one set of maps has much higher variability than the other, and that's why the paired t-test result is largely inflated.

The scaling by sqrt(SOS) artificially equalizes the variance between the two sets of maps and forces the data to meet the sphericity assumption. In the end it renders a more reasonable t-test result. And your simulations basically indicate the same thing. So again if you're not worried about the interpretation, you are doing fine.

Gang



Edited 2 time(s). Last edit at 08/08/2013 10:29AM by Gang.