I've not seen these problems on the Mac, but then I don't use it much. I'll look into it a little more.
The quality setting for the cjpeg program (used to write JPEGs from AFNI) is "95", which is the highest recommended. So there shouldn't be too much loss from the compression - although there is some, particularly at edges in blue-ish colors.
The BMP problem appears to be with the ppmtobmp program, which apparently can produce illegal BMP files. I have modified the source code of AFNI to use ppmtobmp with the "-bpp 24" option, which will skip the color quantization step. This code is now available, but to use it you'll have to recompile the AFNI package -- I'm not putting binaries up just now since I'm making some other changes for the next version and don't want to recompile all the binary packages at this moment.
For TIFF: AFNI uses program "ppm2tiff" preferentially, or "pnmtotiff". On my Mac, I have ppm2tiff installed, so it uses that. I also added the "-c none" (no compression) flag to this. I don't know if this will help on your system. It doesn't make a difference on mine.
Oh, I see you are using "pnmtotiff", but it isn't loading. Well, that's an installation problem (probably from fink?). I have the library file "/sw/lib/libtiff.3.dylib" on my system. Perhaps you should use fink to install libtiff and see if that helps.
bob cox