There are two assumptions in ANOVA regarding factor levels:
(1) Probability distributions of all factor level are normal with the same variance.
(2) The percent signal changes for each factor level are random selections from the corresponding probability distribution and are
independent of the percent signal changes for any other factor level.
If A-B is included as another factor level besides A and B, it would violate assumption (2) above. Looking from another perspective, ANOVA can be viewed as regressoin analysis if factors are coded into indictor (dummy) variables. If A-B is explicitly added as a variable in the regression model, the design matrix X will end up with a non-invertible (X'X).
In your case 3dANOVA2 doesn't really know that the third level of the factor is A-B. Instead it just treats it as an extra level, and that is why it can still give out some results. The fact that you obtained
better results is only interpreted by humans, but cannot be justified from the viewpoint of statistical analysis.
Hope this helps,
Gang
