Hi Jed,
I might miss something, but in your case it seems still reasonable to run group analysis with 3dANOVA in AFNI. Although the behavioral outcome (beta constrast) is estimated differently with different number of trials in each category across subject, theoretically the outcome from each subject is drawn independently from a normal distribution of the behavior in that subject. You might say that the variance (or standard deviation) of the contrast for each subject is estimated differently in terms of pattern (or design matrix) and degree of freedom (or number of trials), but all the assumptions required in 3dANOVA (i.i.d) are still met in the situation.
Anyway, I think both ways you laid out for obtaining standard deviation map from AFNI are correct. In your first approach, other than using '-nodata' option to get (X'X)^(-1) as a separate step, you could also add option '-xout' in 3dDeconvolve in your first step with '-vout', which would spill out the design matrix X and save a separate run for each subject. But your 2nd approach is definitely more straightforward.
Gang