I have modified the source code of 3dABoverlap so that if an input dataset is already in byte format, and has only 1 sub-brick (as produced by 3dAutomask), then the automask operation will not be carried out.
Note that doing a similar operation in two different ways:
3dAutomask -prefix Amm A+orig
3dAutomask -prefix Bmm B+orig
3dABoverlap Amm+orig Bmm+orig
versus
3dAboverlap A+orig B+orig
will probably not give identical results if the datasets A+orig and B+orig are not on the same grid (e.g., anat and EPI), since in the first case, the automask of B is generated and then resampled to the grid of A (inside 3dABoverlap), whereas in the second case, B is resampled to the grid of A and THEN its automask is generated. The different order of resampling and automasking will give different results. Not wildly different, but not identical.