12.15.1. How to use FS recon-all with AFNI¶
Introduction¶
Download script: fs_fsprep.tcsh
FreeSurfer (FS) provides a number of useful tools for brain imaging.
In particular, the parcellation/segmentations and anatomical surfaces
generated by recon-all can be used in lots of applications.
Here we describe using FS’s recon-all, and then bringing its
results into AFNI/SUMA-land. (In this case, we are using FS
ver=7.1.1, but it should work equivalently for most earlier versions.)
Start-to-finish FS example¶
This a compact example of going through the dataset check and running
FS. It is, in fact, what is run on the AFNI Bootcamp data example (on
the anatomical in AFNI_data6/FT_analysis/FT/).
First, we copy the dset to be in NIFTI format (if it isn’t already)
using 3dcopy. Then we run FS’s recon-all to estimate
surfaces, tissue maps and specific anatomical parcellations (and in
this example we assume that the data came from a 3T scanner– hence
the use of the -3T flag). To bring that output into standard
NIFTI and GIFTI format, as well as to generate standard surfaces and
other niceties, we run AFNI’s @SUMA_Make_Spec_FS:
#!/bin/tcsh
# 0) Copy data to NIFTI format (if necessary):
3dcopy FT_anat+orig.HEAD FT_anat_cp.nii.gz
# 1) Run FreeSurfer, basic example A.
recon-all \
-all \
-3T \
-sd . \
-subjid FT \
-i FT_anat_cp.nii.gz
# 2) Import FS results into SUMA-land (and standardize surfaces).
@SUMA_Make_Spec_FS \
-fs_setup \
-NIFTI \
-sid FT \
-fspath ./FT
And that is all. Note that recon-all will take a long time to run
(several hours). There are some ways to speed it up a bit using its
internal parallelization, which you can read about in the next section.
recon-all. We are not so familiar with them, but a
full list for investigating is here:
), then you will
likely have to use additional considerations. For information on
these, read here:Run recon-all faster: -parallel¶
From the FS documentation, there has
been internal parallelization with parts of recon-all since v5.3,
using OpenMP (which is also what several AFNI programs use for
parallelization speedup). You can/should read more about the details
from the FS documentation, but we describe using it here.
At least in the most recent version FS (v7.*), you can add a
-parallel option flag at the end of your recon-all command to
take advantage of a default amount of 4 CPUs. So, going from the
first example, you could run:
# example B: using default parallelization
recon-all \
-all \
-3T \
-sd . \
-subjid FT \
-i FT_anat_cp.nii.gz \
-parallel
Additionally, you should have further control by adding an option
-openmp .., whose single argument is the number of CPUs for OpenMP
to use. Theoretically, this can be more than 4, if you have the
computing power available. So, you could try:
# example C: using parallelization with 8 CPUs
recon-all \
-all \
-3T \
-sd . \
-subjid FT \
-i FT_anat_cp.nii.gz \
-parallel \
-openmp 8
Anecdote 1: on Linux desktop¶
As an anecdote (each of these is a single implementation, not the
result of averaging a set of them), I ran each of the above
recon-all cases on my desktop for the same Bootcamp dataset
described above. This desktop is a modern Ubuntu 20.04 Linux machine
with 20 cores (#humblebrag). In each case, I had 16 threads
available (I had set setenv OMP_NUM_THREADS 16 in my tcsh script).
The recon-all timing results were as follows:
Ex A: 3.751 hours
Ex B: 2.160 hours
Ex C: 1.944 hours
So, using the -parallel option does seem to help speed things
up noticeably (by a bit under a factor of 2, here). Using the
-openmp 8 on top of this did not seem to matter much.
And note: I also ran Ex. A above with setenv OMP_NUM_THREADS 1,
and the runtime was a very similar 3.754 hours. So, if you are not
using -parallel, you might as well just use a single thread—you
don’t get any speedup from OpenMP without that option being used.
Anecdote 2: on Biowulf cluster¶
As another anecdote, I ran each of the above recon-all cases on
the NIH’s Biowulf cluster, for the same Bootcamp dataset described
above. In the parallel cases, I actually had 8 CPUs available (I
requested 8 CPUs from the cluster, and running afni_check_omp in
the terminal indeed returned the value of 8). The recon-all
timing results were as follows:
Ex A: 9.181 hours
Ex B: 5.120 hours
Ex C: 5.093 hours
So, using the -parallel option does seem to help significantly
speed things up (by about a factor of 2, here). I did not get
further benefit by trying to increase the number of threads by also
including the -openmp .. option—I am not sure why. If you are
able to get further runtime improvement somehow, please let us know
how!
Anecdote 3: caveats with -parallel¶
However, please also note: when processing a group of several
hundred anatomical volumes on the cluster, I had several recon-all
runs fail when using the -parallel option. The specific failure
that occurred was this message:
Cannot find rh.white.H
From searching online, apparently this is a known issue that can occur, related to something with the inner workings of the parallelization. I had the same error occur even when running on my desktop once.
So, if this pops up while you are using the -parallel option, try
removing it and rerunning your job. (I had no subsequent failures on
the cluster once I had done this.)
A note of filenames/paths with FS¶
Here we describe how to specify and link together output paths for
running recon-all and @SUMA_Make_Spec_FS.
By default, FS’s recon-all will put its output directory in a
location specified with a $SUBJECTS_DIR environment variable
created at setup. For example, on my computer echo $SUBJECTS_DIR
displayed /usr/local/freesurfer/subjects. However, I much prefer
to specify my own path/location, and hence I use the -sd ..
option.
Consider the following command:
recon-all \
-all \
-3T \
-sd AAA \
-subjid BBB \
-i DSET.nii.gz
After this, the path to the top of the output directory would be:
AAA/BBB/. And to bring the FS output into AFNI/SUMA-land, we could
run:
@SUMA_Make_Spec_FS \
-fs_setup \
-NIFTI \
-sid BBB \
-fspath AAA/BBB
... and the outputs of interest would be in the AAA/BBB/SUMA/
directory. Note how we use the subject ID “BBB” twice: it is required
as part of the path, but we use it optionally after -sid .., so
that various filenames contain it.
These conventionalities were used in the above start-to-finish example. But since we get paid by the word, we thought we would describe such things in more explicit and general and technical and detailed detail here.
A general tcsh script for FS+SUMA¶
Putting this altogether, if we were writing a script to combine
running recon-all and @SUMA_Make_Spec_FS, the following is
probably what The Royal We would do (with tcsh syntax). The first
four variables at the top would be set with our specific file names
and folder locations of choice. After that, everything is automatic,
including saving the terminal text to log files, just in case we want
to check back on things later (and note that recon-all here
includes the -parallel option – whether you want to include that
depends on your system):
#!/bin/tcsh
set dset = INPUT_DSET
set subj = SUBJECT_ID
set dir_fs = PATH_TO_FS_OUTPUT
set dir_echo = PATH_TO_SAVE_STDERR_OUTPUT # maybe: "."
# ------ setup and/or check number of threads
### can uncomment next line if this should be set here (NB: I am
### aiming to use 4 threads below in recon-all with the '-parallel opt)
# setenv OMP_NUM_THREADS 4
set nomp = `afni_check_omp`
echo "++ Should be using this many threads: ${nomp}" \
> ${dir_echo}/o.00_fs_${subj}.txt
# ------ run programs, logging terminal output and exiting on failure
\mkdir -p ${dir_fs}
time recon-all \
-all \
-3T \
-sd ${dir_fs} \
-subjid ${subj} \
-i ${dset} \
-parallel \
|& tee -a ${dir_echo}/o.00_fs_${subj}.txt
if ( $status ) then
echo "** ERROR running FS recon-all for: ${subj}" \
|& tee -a ${dir_echo}/o.00_fs_${subj}.txt
exit 1
endif
@SUMA_Make_Spec_FS \
-fs_setup \
-NIFTI \
-sid ${subj} \
-fspath ${dir_fs}/${subj} \
|& tee ${dir_echo}/o.01_suma_makespec_${subj}.txt
if ( $status ) then
echo "** ERROR running @SUMA_Make_Spec_FS for: ${subj}" \
|& tee -a ${dir_echo}/o.01_suma_makespec_${subj}.txt
exit 1
endif
echo "++ Done with FS + conversion to SUMA for: ${subj}"
The main FS output would be in ${dir_fs}/${subj}/, and the
converted NIFTI/GIFTI files to carry on with would be in
${dir_fs}/${subj}/SUMA/.
The above could be translated to a bash script, just changing the
syntax in lines with setenv and set, as well as the way
teeing is done.
A note on @SUMA_Make_Spec_FS outputs¶
The final SUMA/ directory contains: volumetric outputs of
segmentations and parcellations, surfaces of various sizes and
geometry, and more. Several of these data sets are direct copies of
FS output, but in NIFTI and other formats usable by AFNI. We also
generate standardized surfaces, which are very useful for group
analysis, and you can read more about that here:
https://pubmed.ncbi.nlm.nih.gov/16035046/
We also derive some other datasets that we have found to be useful, such as groupings of parcellated ROIs by tissue types. These helpful datasets, stats files and QC images are now described in detail here.
Minor note on FS setup¶
By default, after you have set up FreeSurfer, every time you open a
new terminal or source one of your ~/.*rc files, you will get some
text about your FS setup displayed in the terminal. This comes from
the FS setup script that is run each time, and looks something like:
-------- freesurfer-linux-centos7_x86_64-7.1.1-20200723-8b40551 --------
Setting up environment for FreeSurfer/FS-FAST (and FSL)
FREESURFER_HOME /usr/local/freesurfer
FSFAST_HOME /usr/local/freesurfer/fsfast
FSF_OUTPUT_FORMAT nii.gz
SUBJECTS_DIR /usr/local/freesurfer/subjects
MNI_DIR /usr/local/freesurfer/mni
The exact text varies based on your OS, version of FS, location of the binaries, etc.
Anyways, if you would like to disable the display of that text message, you can do the following:
For
bashshell users, put the following into your~/.bashrcfile:export FS_FREESURFERENV_NO_OUTPUT="OFF"
... above the
source $FREESURFER_HOME/SetUpFreeSurfer.shline.For
tcshshell users, put the following into your~/.cshrcfile:setenv FS_FREESURFERENV_NO_OUTPUT "OFF"
... above the
source $FREESURFER_HOME/SetUpFreeSurfer.cshline.
If you open a new terminal, you should not see the setup info text, but you should still be able to run FS programs fine.
This is, of course, entirely optional.
