We found the data to have incorrect header information for both the markers in the anat+orig dataset and for the transformation matrices in the anat+tlrc dataset. Redoing the marker points and recomputing the manual transformation did produce a useful transformation. None of us have seen this problem before. Is there a chance the .HEAD files have been manually edited? As the best solution for now, you can redo the transformations for those two subjects or use the @auto_tlrc transformation for all subjects (not just the two that have problems).
This is most likely just a typo, but the dilate parameter in your 3dAutomask command should be at least 1 to be useful; otherwise, you can just leave the dilate option out.
