Suppose you want to erode the edges; that is, voxels that are in the mask but whose neighbors are not all in the mask. This could be done with
3dcalc -a mask+orig \
-b a+i -c a-i -d a+j -e a-j -f a+k -g a-k \
-expr 'step(a)*mofn(6,b,c,d,e,f,g)' -prefix maskminus -nscale
Here, the mofn(6,b,c,d,e,f,g) function is 1 if all 6 arguments after the '6' are nonzero, and is 0 otherwise. So if 'a' is 1 but any of its neighbors are 0, then that voxel is eroded. If you want to allow 'a' to survive if (say) 5 of its neighbors are in, then change the '6' to a '5'.
As always, the easiest way to try out new types of 3dcalc expressions is via using the ccalc program.