> I was wondering if at this stage it would be appropriate to replace the second-level generated mask (mask.nii)
> with a grey matter only or ROI mask and still get accurate smoothing estimates?

If the quality of the gray matter or ROI mask is reasonable and you can justify its usage for group analysis, it's probably fine. On the other hand, since the estimation of spatial relatedness is an approximation step anyway, I would simply use the whole brain mask to estimate the spatial relatedness. A smaller mask may compromise the estimation accuracy as you've already found out. The reason could be that a smaller mask may distort the estimation especially when the mask contains narrow and tortuous regions.

Gang