How did you fractionalize the mask from +tlrc coordinates to +orig coordinates - by using the "-warp" option of 3dfractionize? Did you also use the "-preserve" option? Also, you might need to use the "-clip" option, otherwise voxels in the +orig dataset that only very slightly overlap the +tlrc ROI will become part of the ROI, which will cause the ROI to expand.
You should look at the ROI in +tlrc and +orig space to see what the geometrical or numerical cause of the problem is.
bob cox