14.2.1. Reynolds et al. (2024). Processing, evaluating and understanding FMRI data with afni_proc.py¶
Introduction¶
Here we present commands used in the following paper:
- Reynolds RC, Glen DR, Chen G, Saad ZS, Cox RW, Taylor PA (2024). Processing, evaluating and understanding FMRI data with afni_proc.py. Imaging Neuroscience 2:1-52.
Abstract: FMRI data are noisy, complicated to acquire, and typically go through many steps of processing before they are used in a study or clinical practice. Being able to visualize and understand the data from the start through the completion of processing, while being confident that each intermediate step was successful, is challenging. AFNI’s afni_proc.py is a tool to create and run a processing pipeline for FMRI data. With its flexible features, afni_proc.py allows users to both control and evaluate their processing at a detailed level. It has been designed to keep users informed about all processing steps: it does not just process the data, but also first outputs a fully commented processing script that the users can read, query, interpret, and refer back to. Having this full provenance is important for being able to understand each step of processing; it also promotes transparency and reproducibility by keeping the record of individual-level processing and modeling specifics in a single, shareable place. Additionally, afni_proc.py creates pipelines that contain several automatic self-checks for potential problems during runtime. The output directory contains a dictionary of relevant quantities that can be programmatically queried for potential issues and a systematic, interactive quality control (QC) HTML. All of these features help users evaluate and understand their data and processing in detail. We describe these and other aspects of afni_proc.py here using a set of task-based and resting-state FMRI example commands.
Study keywords: FMRI, EPI, MPRAGE, pipeline, reproducibility, understanding, quality control, visualization, multi-echo, surface processing
Main programs:
afni_proc.py, sswarper2, open_apqc.py,
ap_run_simple_rest.tcsh, ap_run_simple_rest_me.tcsh
Github page:
See these authors’ github page for descriptions and downloads
of codes and supplementary text files:
Download scripts¶
Since there are several scripts, we recommend downloading directly from GitHub, such as by copy+pasting this into a terminal:
git clone https://github.com/afni/apaper_afni_proc.git
View scripts¶
These are just a couple examples of the “Desktop” scripts on GitHub, please see above for downloading all scripts (both the “Desktop” and “Biowulf” versions).
do_20_ap_simple.tcsh¶
This script contains the “simple” afni_proc.py wrapper for ME-FMRI processing.
1#!/bin/tcsh
2
3# AP simple: run afni_proc.py for full FMRI processing (for initial QC)
4# -> run the ME-FMRI simple command here
5
6# Process a single subj+ses pair.
7
8# This is a Desktop script. Run it via swarm (see partner run*.tcsh).
9
10
11# initial exit code; we don't exit at fail, to copy partial results back
12set ecode = 0
13
14# ---------------------------------------------------------------------------
15# top level definitions (constant across demo)
16# ---------------------------------------------------------------------------
17
18# labels
19set subj = $1
20set ses = $2
21set ap_label = 20_ap_simple
22
23
24# upper directories
25set dir_inroot = ${PWD:h} # one dir above scripts/
26set dir_log = ${dir_inroot}/logs
27set dir_basic = ${dir_inroot}/data_00_basic
28set dir_fs = ${dir_inroot}/data_12_fs
29set dir_ssw = ${dir_inroot}/data_13_ssw
30set dir_physio = ${dir_inroot}/data_14_physio
31set dir_ap = ${dir_inroot}/data_${ap_label}
32
33# subject directories
34set sdir_basic = ${dir_basic}/${subj}/${ses}
35set sdir_func = ${sdir_basic}/func
36set sdir_anat = ${sdir_basic}/anat
37set sdir_fs = ${dir_fs}/${subj}/${ses}
38set sdir_suma = ${sdir_fs}/SUMA
39set sdir_ssw = ${dir_ssw}/${subj}/${ses}
40set sdir_physio = ${dir_physio}/${subj}/${ses}
41set sdir_ap = ${dir_ap}/${subj}/${ses}
42
43# supplementary directories and info
44set dir_suppl = ${dir_inroot}/supplements
45set template = ${dir_suppl}/MNI152_2009_template_SSW.nii.gz
46
47# set output directory
48set sdir_out = ${sdir_ap}
49set lab_out = AP
50
51# --------------------------------------------------------------------------
52# data and control variables
53# --------------------------------------------------------------------------
54
55setenv AFNI_COMPRESSOR GZIP
56
57# dataset inputs
58set task_label = task-rest_run-1
59
60set epi_radix = ${sdir_func}/${subj}_${ses}
61set dset_epi_e2 = ( ${epi_radix}_${task_label}_echo-2_bold.nii* )
62set dsets_epi_me = ( ${epi_radix}_${task_label}_echo-?_bold.nii* )
63set me_times = ( 12.5 27.6 42.7 )
64
65set dset_anat_00 = ${sdir_anat}/${subj}_${ses}_mprage_run-1_T1w.nii.gz
66
67# control variables
68set nt_rm = 4 # number of time points to remove at start
69#set blur_size = 6 # blur size to apply
70#set final_dxyz = 3 # final voxel size (isotropic dim)
71#set cen_motion = 0.2 # censor threshold for motion (enorm)
72#set cen_outliers = 0.05 # censor threshold for outlier frac
73
74
75# check available N_threads and report what is being used
76set nthr_avail = `afni_system_check.py -disp_num_cpu`
77set nthr_using = `afni_check_omp`
78
79echo "++ INFO: Using ${nthr_using} of available ${nthr_avail} threads"
80
81
82# ---------------------------------------------------------------------------
83# run programs
84# ---------------------------------------------------------------------------
85
86# make output directory and go to it
87\mkdir -p ${sdir_out}
88cd ${sdir_out}
89
90# create command script
91set run_script = ap.cmd.${subj}
92
93cat << EOF >! ${run_script}
94
95# AP, run simple
96#
97# multi-echo FMRI, simple processing for initial QC
98# anatomical has skull on
99
100ap_run_simple_rest_me.tcsh \
101 -run_ap \
102 -subjid ${subj} \
103 -nt_rm ${nt_rm} \
104 -anat ${dset_anat_00} \
105 -epi_me_run ${dsets_epi_me} \
106 -echo_times ${me_times} \
107 -template ${template}
108
109EOF
110
111if ( ${status} ) then
112 set ecode = 1
113 goto COPY_AND_EXIT
114endif
115
116
117# execute AP command to make processing script
118tcsh -xef ${run_script} |& tee output.ap.cmd.${subj}
119
120if ( ${status} ) then
121 set ecode = 2
122 goto COPY_AND_EXIT
123endif
124
125
126# execute the proc script, saving text info
127time tcsh -xef proc.${subj} |& tee output.proc.${subj}
128
129if ( ${status} ) then
130 set ecode = 3
131 goto COPY_AND_EXIT
132endif
133
134echo "++ FINISHED ${lab_out}"
135
136# ---------------------------------------------------------------------------
137
138COPY_AND_EXIT:
139
140
141if ( ${ecode} ) then
142 echo "++ BAD FINISH: ${lab_out} (ecode = ${ecode})"
143else
144 echo "++ GOOD FINISH: ${lab_out}"
145endif
146
147exit ${ecode}
do_23_ap_ex3_vol.tcsh¶
1#!/bin/tcsh
2
3# AP-3: run afni_proc.py for full FMRI processing (Example 3)
4
5# Process a single subj+ses pair.
6
7# This is a Desktop script. Run it via swarm (see partner run*.tcsh).
8
9
10# initial exit code; we don't exit at fail, to copy partial results back
11set ecode = 0
12
13# ---------------------------------------------------------------------------
14# top level definitions (constant across demo)
15# ---------------------------------------------------------------------------
16
17# labels
18set subj = $1
19set ses = $2
20set ap_label = 23_ap_ex3_vol
21
22
23# upper directories
24set dir_inroot = ${PWD:h} # one dir above scripts/
25set dir_log = ${dir_inroot}/logs
26set dir_basic = ${dir_inroot}/data_00_basic
27set dir_fs = ${dir_inroot}/data_12_fs
28set dir_ssw = ${dir_inroot}/data_13_ssw
29set dir_physio = ${dir_inroot}/data_14_physio
30set dir_ap = ${dir_inroot}/data_${ap_label}
31
32# subject directories
33set sdir_basic = ${dir_basic}/${subj}/${ses}
34set sdir_func = ${sdir_basic}/func
35set sdir_anat = ${sdir_basic}/anat
36set sdir_fs = ${dir_fs}/${subj}/${ses}
37set sdir_suma = ${sdir_fs}/SUMA
38set sdir_ssw = ${dir_ssw}/${subj}/${ses}
39set sdir_physio = ${dir_physio}/${subj}/${ses}
40set sdir_ap = ${dir_ap}/${subj}/${ses}
41
42# supplementary directories and info
43set dir_suppl = ${dir_inroot}/supplements
44set template = ${dir_suppl}/MNI152_2009_template_SSW.nii.gz
45
46# set output directory
47set sdir_out = ${sdir_ap}
48set lab_out = AP
49
50# --------------------------------------------------------------------------
51# data and control variables
52# --------------------------------------------------------------------------
53
54setenv AFNI_COMPRESSOR GZIP
55
56# dataset inputs
57set task_label = task-rest_run-1
58
59set epi_radix = ${sdir_func}/${subj}_${ses}
60set dset_epi = ( ${epi_radix}_${task_label}_echo-2_bold.nii* ) # 2nd echo
61set dsets_epi_me = ( ${epi_radix}_${task_label}_echo-?_bold.nii* )
62set me_times = ( 12.5 27.6 42.7 )
63
64set blip_radix = ${sdir_func}/${subj}_${ses}_acq-blip
65set epi_forward = "${blip_radix}_dir-match_run-1_bold.nii.gz[0]"
66set epi_reverse = "${blip_radix}_dir-opp_run-1_bold.nii.gz[0]"
67
68set physio_radix = ${sdir_physio}/${subj}_${ses}
69set physio_regs = ${physio_radix}_${task_label}_physio_slibase.1D
70
71set anat_cp = ( ${sdir_ssw}/anatSS.${subj}.nii* )
72set anat_skull = ( ${sdir_ssw}/anatU.${subj}.nii* )
73
74set dsets_NL_warp = ( ${sdir_ssw}/anatQQ.${subj}.nii* \
75 ${sdir_ssw}/anatQQ.${subj}.aff12.1D \
76 ${sdir_ssw}/anatQQ.${subj}_WARP.nii* )
77
78set roi_all_2009 = ${sdir_suma}/aparc.a2009s+aseg_REN_all.nii.gz
79set roi_gmr_2009 = ${sdir_suma}/aparc.a2009s+aseg_REN_gmrois.nii.gz
80set roi_gmr_2000 = ${sdir_suma}/aparc+aseg_REN_gmrois.nii.gz
81set roi_FSvent = ${sdir_suma}/fs_ap_latvent.nii.gz
82set roi_FSWe = ${sdir_suma}/fs_ap_wm.nii.gz
83
84# control variables
85set nt_rm = 4 # number of time points to remove at start
86set blur_size = 5 # blur size in mm, usually based of voxel size
87set final_dxyz = 3 # final voxel size (isotropic dim)
88set cen_motion = 0.2 # censor threshold for motion (enorm)
89set cen_outliers = 0.05 # censor threshold for outlier frac
90
91
92# check available N_threads and report what is being used
93set nthr_avail = `afni_system_check.py -disp_num_cpu`
94set nthr_using = `afni_check_omp`
95
96echo "++ INFO: Using ${nthr_using} of available ${nthr_avail} threads"
97
98
99# ---------------------------------------------------------------------------
100# run programs
101# ---------------------------------------------------------------------------
102
103# make output directory and go to it
104\mkdir -p ${sdir_out}
105cd ${sdir_out}
106
107# create command script
108set run_script = ap.cmd.${subj}
109
110cat << EOF >! ${run_script}
111
112# AP, Example 3: for voxelwise analysis
113#
114# single echo FMRI
115# volumetric, voxelwise analysis, warped to standard space
116# include physio regressors
117# include follower GM-ROIs from FS 2009 parc
118
119# we do NOT include bandpassing here (see comments in text)
120
121afni_proc.py \
122 -subj_id ${subj} \
123 -dsets ${dset_epi} \
124 -copy_anat ${anat_cp} \
125 -anat_has_skull no \
126 -anat_follower anat_w_skull anat ${anat_skull} \
127 -anat_follower_ROI aagm09 anat ${roi_gmr_2009} \
128 -anat_follower_ROI aegm09 epi ${roi_gmr_2009} \
129 -blocks ricor tshift align tlrc volreg mask blur \
130 scale regress \
131 -radial_correlate_blocks tcat volreg regress \
132 -tcat_remove_first_trs ${nt_rm} \
133 -ricor_regs ${physio_regs} \
134 -ricor_regs_nfirst ${nt_rm} \
135 -ricor_regress_method per-run \
136 -align_unifize_epi local \
137 -align_opts_aea -cost lpc+ZZ -giant_move -check_flip \
138 -tlrc_base ${template} \
139 -tlrc_NL_warp \
140 -tlrc_NL_warped_dsets ${dsets_NL_warp} \
141 -volreg_align_to MIN_OUTLIER \
142 -volreg_align_e2a \
143 -volreg_tlrc_warp \
144 -volreg_warp_dxyz ${final_dxyz} \
145 -volreg_compute_tsnr yes \
146 -mask_epi_anat yes \
147 -blur_size ${blur_size} \
148 -regress_motion_per_run \
149 -regress_make_corr_vols aegm09 \
150 -regress_censor_motion ${cen_motion} \
151 -regress_censor_outliers ${cen_outliers} \
152 -regress_apply_mot_types demean deriv \
153 -regress_est_blur_epits \
154 -regress_est_blur_errts \
155 -html_review_style pythonic
156
157EOF
158
159if ( ${status} ) then
160 set ecode = 1
161 goto COPY_AND_EXIT
162endif
163
164
165# execute AP command to make processing script
166tcsh -xef ${run_script} |& tee output.ap.cmd.${subj}
167
168if ( ${status} ) then
169 set ecode = 2
170 goto COPY_AND_EXIT
171endif
172
173
174# execute the proc script, saving text info
175time tcsh -xef proc.${subj} |& tee output.proc.${subj}
176
177if ( ${status} ) then
178 set ecode = 3
179 goto COPY_AND_EXIT
180endif
181
182echo "++ FINISHED ${lab_out}"
183
184# ---------------------------------------------------------------------------
185
186COPY_AND_EXIT:
187
188
189if ( ${ecode} ) then
190 echo "++ BAD FINISH: ${lab_out} (ecode = ${ecode})"
191else
192 echo "++ GOOD FINISH: ${lab_out}"
193endif
194
195exit ${ecode}
