To fix the 3dfractionize error, you can use 3dcalc to convert a dataset to byte or short format.
3dcalc -byte -prefix AMG_mask_byte \
-a ROImask_BilatAmy+tlrc \
-expr 'a'
The data not ending up in the right place was due to not specifying a warp to go from the standard space mask to the subject space residual data. Providing 3dfractionize with -warp SubjectAnatomical+tlrc will work if you used @auto_tlrc to normalize the data.
Though for your data, if you want to use 3dttest++ and you have processed your data in subject (+orig) space, then I would suggest putting both the stats dataset and the residual dataset into standard space (+tlrc) via @auto_tlrc or adwarp. Once everything in in standard space, you can use 3dFWHMx to estimate the smoothness of the residual for each subject in standard space, and use those values as the input for 3dClustSim's -fwhmxyz argument. You may also need to use 3dClustSim's -OKsmallmask argument.
3dFWHMx -mask ROImask_BilatAmy+tlrc -detrend -input subject1_residual+tlrc
3dFWHMx -mask ROImask_BilatAmy+tlrc -detrend -input subject2_residual+tlrc
...
Let me/us know if that's more clear!
Edited 3 time(s). Last edit at 09/29/2014 10:13PM by Peter Molfese.
