After changing those two variables, running the anova, I still encountered the same error. A little over 30 minutes to get to slice 256 before seeing:

*** caught segfault ***
address 0x15f3e5f030, cause 'memory not mapped'

Traceback:
1: write.c.AFNI(filename, dset = brk, label = label, space = space, note = note, origin = origin, delta = delta, orient = orient, idcode = idcode, defhead = defhead, verb = verb, maskinf = maskinf, scale = scale, addFDR = addFDR, statsym = statsym, view = view, com_hist = com_hist, type = type, TR = TR, overwrite = overwrite)
2: write.AFNI(lop$outFN, out, brickNames, defhead = head, idcode = newid.AFNI(), com_hist = lop$com_history, statsym = statsym, addFDR = 1, type = "MRI_short", overwrite = lop$overwrite)
An irrecoverable exception occurred. R is aborting now ...
Segmentation fault

I should note before I was encountering this error, I was getting the "vector memory limit reached" but when i changed the R_MAX_VSIZE variable to 200Gb, and I no longer got that error. I'm thinking that the memory limit set by R actually includes both the total memory used and the compressed memory, and when the final slice is done, the memory and compressed memory combined go over 200 (meanwhile the memory pressure visual doesnt indicate anything is having a problem). Im on a mac, high Sierra, 32gb DDR4, 4.2 GHz Intel Core i7. Any ideas?