Hi Chris,

We generally don't rotate DTI tensor values, only the positions, AFAIK. As Rick points out, it's problematic, and it probably won't be really right for a nonlinear alignment problems, which is how people would generally do an aligment nowadays.

We do take care of the somewhat related problem of transforming gradient vectors (1D lists of xyz vectors) for data that has to be rotated for motion correction or for "axialization" to a template. One could compute the eigenvectors from the tensor and rotate each of those. Then recompute the tensor from those transformed eigenvectors. Seems some trouble though. The use case seems a bit difficult to see. Do you need to transform the tensor or maybe something else like the primary eigenvector or some other aspect of the DTI atlas? That atlas has FA, primary eigenvectors and a number of other common DTI maps that could be transformed.

In case you do want to pursue something for this, consider programs like Vecwarp for applying affine transformations to lists of xyz coordinates.

I don't know much about the AFNI-ANTs transformation conversion, but combining different definitions of affine warps has been described here:
[neurostars.org]
If you try it, let me know how it works out.

Nonlinear transformations, surprisingly, should be easier to apply because they are a simpler dx,dy,dz at each voxel by most software packages. Of course, the direction of each package may be different, and there can be issues in the gridding of input and output, so one has to approach that with care too.