afni.nimh.nih.gov

Hi Rick,

If you have time, I know one user (me), who would really appreciate some (hopefully easy) changes to some of the afni_proc.py options.

I'm trying to get into the habit of running -afni_proc.py for each new study, and anytime I make changes in processing stream, rather than edit old ones. This way I stay tuned in to the latest and greatest going-ons over at AFNI.

Anyway, I'm one of those crazies that would prefer to use underscore '_' instead of '.' as the separation character in the filenames. And all these issues stem from that one problem.

-sep_char CHAR : apply as separation character in filenames

...

If (for some crazy reason) an underscore (_) character would be
preferable, the result would be:

If "-sep_char _" is applied, so is -subj_curly.

See also -subj_curly.

There seem to be some files other than transformation matrix 1D files and rm. files that keep the '.' separation.

For my purposes - this is the problem bit (I'll include the full output at the end of the message).
>
# ==========================================================================
# script generated by the command:
#
# afni_proc.py -script Single-Preprocessing.sh -scr_overwrite -out_dir \
# '${subj}' -blocks tshift align tlrc volreg blur mask scale -copy_anat \
# /Volumes/BuckleyProjects/RAWorganization/Imaging/MBMF/MPRage160sl7degp2SC01821.nii.gz \
# -dsets \
# /Volumes/BuckleyProjects/RAWorganization/Imaging/MBMF/BOLDRUN1SC01821A.nii.gz \
# /Volumes/BuckleyProjects/RAWorganization/Imaging/MBMF/BOLDRUN2SC01821A.nii.gz \
# /Volumes/BuckleyProjects/RAWorganization/Imaging/MBMF/BOLDRUN3SC01821A.nii.gz \
# /Volumes/BuckleyProjects/RAWorganization/Imaging/MBMF/BOLDRUN4SC01821A.nii.gz \
# -volreg_align_to first -volreg_interp -heptic -volreg_opts_vr \
# -maxdisp1D 'md_r${run}.1D' -1Dfile 'motion_r${run}}.1D' \
# -volreg_align_e2a -subj_curly -sep_char _ -blur_size 6 -align_opts_aea \
# -AddEdge -move_preproc_files -tlrc_NL_warp -tlrc_base \
# ~/abin/MNI152_1mm_uni+tlrc -blur_to_fwhm \
# -volreg_tlrc_warp
>

Issue with option -out_dir:
In the verification step, it prints out the old format (not an issue, but aesthetically displeasing)
>
# assign output directory name
set output_dir = ${subj}

# verify that the results directory does not yet exist
if ( -d $output_dir ) then
echo output dir "$subj.results" already exists
exit
endif
>

Issue with sep_char
individual outcount runs
>
# ========================== auto block: outcount ==========================
# data check: compute outlier fraction for each volume
touch out.pre_ss_warn.txt
foreach run ( $runs )
3dToutcount -automask -fraction -polort 4 -legendre \
pb00_${subj}_r${run}_tcat+orig > outcount.r$run.1D

# outliers at TR 0 might suggest pre-steady state TRs
if ( `1deval -a outcount.r$run.1D"{0}" -expr "step(a-0.4)"` ) then
echo "** TR #0 outliers: possible pre-steady state TRs in run $run" \
>> out.pre_ss_warn.txt
endif
end

# catenate outlier counts into a single time series
cat outcount.r*.1D > outcount_rall.1D
>

Similarly w/ the volreg dfile (but I can get around that with adding a -volreg_opts_vr -1Dfile 'motion_r${run}}.1D')
>
# register and warp
foreach run ( $runs )
# register each volume to the base
3dvolreg -verbose -zpad 1 -base pb01_${subj}_r01_tshift+orig'[0]' \
-1Dfile dfile.r$run.1D -prefix rm.epi.volreg.r$run \
-heptic \
-maxdisp1D md_r${run}.1D -1Dfile motion_r${run}}.1D \
-1Dmatrix_save mat.r$run.vr.aff12.1D \
pb01_${subj}_r${run}_tshift+orig

...

# make a single file of registration params
cat dfile.r*.1D > dfile_rall.1D
>

Making the final anat file
>
# create an anat_final dataset, aligned with stats
3dcopy MPRage160sl7degp2SC01821_ns+tlrc anat_final.${subj}
>
>

and mask of the anatomy (this also lacks the curly _${subj}
>
# convert to binary anat mask; fill gaps and holes
3dmask_tool -dilate_input 5 -5 -fill_holes -input rm.resam.anat+tlrc \
-prefix mask_anat.$subj
>

Issue with option -subj_curly (mask block, review scripts, and move_preproc_files block)
>
# ================================== mask ==================================
# create 'full_mask' dataset (union mask)
foreach run ( $runs )
3dAutomask -dilate 1 -prefix rm.mask_r$run pb03_${subj}_r${run}_blur+tlrc
end

# get mean and compare it to 0 for taking 'union'
3dMean -datum short -prefix rm.mean rm.mask*.HEAD
3dcalc -a rm.mean+tlrc -expr 'ispositive(a-0)' -prefix full_mask_$subj

# ---- create subject anatomy mask, mask_anat.$subj+tlrc ----
# (resampled from tlrc anat)
3dresample -master full_mask_$subj+tlrc -input \
MPRage160sl7degp2SC01821_ns+tlrc \
-prefix rm.resam.anat

# convert to binary anat mask; fill gaps and holes
3dmask_tool -dilate_input 5 -5 -fill_holes -input rm.resam.anat+tlrc \
-prefix mask_anat.$subj

# compute overlaps between anat and EPI masks
3dABoverlap -no_automask full_mask_$subj+tlrc mask_anat.$subj+tlrc \
|& tee out.mask_overlap.txt

# ---- create group anatomy mask, mask_group+tlrc ----
# (resampled from tlrc base anat, MNI152_1mm_uni+tlrc)
3dresample -master full_mask_$subj+tlrc -prefix ./rm.resam.group \
-input ~/abin/MNI152_1mm_uni+tlrc

# convert to binary group mask; fill gaps and holes
3dmask_tool -dilate_input 5 -5 -fill_holes -input rm.resam.group+tlrc \
-prefix mask_group
>

>
# ================== auto block: generate review scripts ===================

# generate a review script for the unprocessed EPI data
gen_epi_review.py -script @epi_review.$subj \
-dsets pb00_${subj}_r*_tcat+orig.HEAD

# generate scripts to review single subject results
# (try with defaults, but do not allow bad exit status)
gen_ss_review_scripts.py -exit0

# ========================== auto block: finalize ==========================

# remove temporary files
\rm -fr rm.* awpy

# move preprocessing files to 'preproc.data' directory
mkdir preproc.data
mv dfile.r*.1D pb*.$subj.r*.* outcount.* preproc.data

# if the basic subject review script is here, run it
# (want this to be the last text output)
if ( -e @ss_review_basic ) ./@ss_review_basic |& tee out.ss_review.$subj.txt
>

Again the move preprocessing files has both issues.
>
# move preprocessing files to 'preproc.data' directory
mkdir preproc.data
mv dfile.r*.1D pb*.$subj.r*.* outcount.* preproc.data
>

Those are all the issues that I caught with my command, hopefully you will find them useful if you are willing to make those changes.
Here is the full output (plus a quality assurance block that makes a jpg of various parameters I like seeing graphically).

Thanks so much,
Hugo Decker


++++++++++++++
Full Output of script

#!/bin/tcsh -xef

echo "auto-generated by afni_proc.py, Thu Jan 9 14:18:46 2014"
echo "(version 4.06, December 1, 2013)"
echo "execution started: `date`"

# execute via :
# tcsh -xef Single-Preprocessing.sh |& tee output.Single-Preprocessing.sh

# =========================== auto block: setup ============================
# script setup

# take note of the AFNI version
afni -ver

# check that the current AFNI version is recent enough
afni_history -check_date 29 Nov 2013
if ( $status ) then
echo "** this script requires newer AFNI binaries (than 29 Nov 2013)"
echo " (consider: @update.afni.binaries -defaults)"
exit
endif

# the user may specify a single subject to run with
if ( $#argv > 0 ) then
set subj = $argv[1]
else
set subj = SUBJ
endif

# assign output directory name
set output_dir = ${subj}

# verify that the results directory does not yet exist
if ( -d $output_dir ) then
echo output dir "$subj.results" already exists
exit
endif

# set list of runs
set runs = (`count -digits 2 1 4`)

# create results and stimuli directories
mkdir $output_dir
mkdir $output_dir/stimuli

# copy anatomy to results dir
3dcopy \
/Volumes/BuckleyProjects/RAWorganization/Imaging/MBMF/MPRage160sl7degp2SC01821.nii.gz \
$output_dir/MPRage160sl7degp2SC01821

# ============================ auto block: tcat ============================
# apply 3dTcat to copy input dsets to results dir, while
# removing the first 0 TRs
3dTcat -prefix $output_dir/pb00_${subj}_r01_tcat \
/Volumes/BuckleyProjects/RAWorganization/Imaging/MBMF/BOLDRUN1SC01821A.nii.gz'[0..$]'
3dTcat -prefix $output_dir/pb00_${subj}_r02_tcat \
/Volumes/BuckleyProjects/RAWorganization/Imaging/MBMF/BOLDRUN2SC01821A.nii.gz'[0..$]'
3dTcat -prefix $output_dir/pb00_${subj}_r03_tcat \
/Volumes/BuckleyProjects/RAWorganization/Imaging/MBMF/BOLDRUN3SC01821A.nii.gz'[0..$]'
3dTcat -prefix $output_dir/pb00_${subj}_r04_tcat \
/Volumes/BuckleyProjects/RAWorganization/Imaging/MBMF/BOLDRUN4SC01821A.nii.gz'[0..$]'

# and make note of repetitions (TRs) per run
set tr_counts = ( 202 202 202 202 )

# -------------------------------------------------------
# enter the results directory (can begin processing data)
cd $output_dir


# ========================== auto block: outcount ==========================
# data check: compute outlier fraction for each volume
touch out.pre_ss_warn.txt
foreach run ( $runs )
3dToutcount -automask -fraction -polort 4 -legendre \
pb00_${subj}_r${run}_tcat+orig > outcount.r$run.1D

# outliers at TR 0 might suggest pre-steady state TRs
if ( `1deval -a outcount.r$run.1D"{0}" -expr "step(a-0.4)"` ) then
echo "** TR #0 outliers: possible pre-steady state TRs in run $run" \
>> out.pre_ss_warn.txt
endif
end

# catenate outlier counts into a single time series
cat outcount.r*.1D > outcount_rall.1D

# ================================= tshift =================================
# time shift data so all slice timing is the same
foreach run ( $runs )
3dTshift -tzero 0 -quintic -prefix pb01_${subj}_r${run}_tshift \
pb00_${subj}_r${run}_tcat+orig
end

# ================================= align ==================================
# for e2a: compute anat alignment transformation to EPI registration base
# (new anat will be intermediate, stripped, MPRage160sl7degp2SC01821_ns+orig)
align_epi_anat.py -anat2epi -anat MPRage160sl7degp2SC01821+orig \
-save_skullstrip -suffix _al_junk \
-epi pb01_${subj}_r01_tshift+orig -epi_base 0 \
-epi_strip 3dAutomask \
-AddEdge \
-volreg off -tshift off

# ================================== tlrc ==================================
# warp anatomy to standard space (non-linear warp)
auto_warp.py -base ~/abin/MNI152_1mm_uni+tlrc -input \
MPRage160sl7degp2SC01821_ns+orig \
-skull_strip_input no

# move results up out of the awpy directory
# (NL-warped anat, affine warp, NL warp)
# (use typical standard space name for anat)
3dbucket -prefix MPRage160sl7degp2SC01821_ns \
awpy/MPRage160sl7degp2SC01821_ns.aw.nii
mv awpy/anat.un.aff.Xat.1D .
mv awpy/anat.un.aff.qw_WARP.nii .

# ================================= volreg =================================
# align each dset to base volume, align to anat, warp to tlrc space

# verify that we have a +tlrc warp dataset
if ( ! -f MPRage160sl7degp2SC01821_ns+tlrc.HEAD ) then
echo "** missing +tlrc warp dataset: \
MPRage160sl7degp2SC01821_ns+tlrc.HEAD"
exit
endif

# register and warp
foreach run ( $runs )
# register each volume to the base
3dvolreg -verbose -zpad 1 -base pb01_${subj}_r01_tshift+orig'[0]' \
-1Dfile dfile.r$run.1D -prefix rm.epi.volreg.r$run \
-heptic \
-maxdisp1D md_r${run}.1D -1Dfile motion_r${run}}.1D \
-1Dmatrix_save mat.r$run.vr.aff12.1D \
pb01_${subj}_r${run}_tshift+orig

# create an all-1 dataset to mask the extents of the warp
3dcalc -overwrite -a pb01_${subj}_r${run}_tshift+orig -expr 1 \
-prefix rm.epi.all1

# catenate volreg, epi2anat and tlrc transformations
cat_matvec -ONELINE \
anat.un.aff.Xat.1D \
MPRage160sl7degp2SC01821_al_junk_mat.aff12.1D -I \
mat.r$run.vr.aff12.1D > mat.r$run.warp.aff12.1D

# apply catenated xform: volreg, epi2anat and tlrc
# then apply non-linear standard-space warp
3dNwarpApply -master MPRage160sl7degp2SC01821_ns+tlrc -dxyz 3 \
-source pb01_${subj}_r${run}_tshift+orig \
-nwarp anat.un.aff.qw_WARP.nii \
-affter mat.r$run.warp.aff12.1D \
-prefix rm.epi.nomask.r$run

# warp the all-1 dataset for extents masking
3dNwarpApply -master MPRage160sl7degp2SC01821_ns+tlrc -dxyz 3 \
-source rm.epi.all1+orig \
-nwarp anat.un.aff.qw_WARP.nii \
-prefix rm.epi.1.r$run \
-affter mat.r$run.warp.aff12.1D \
-ainterp NN -quiet

# make an extents intersection mask of this run
3dTstat -min -prefix rm.epi.min.r$run rm.epi.1.r$run+tlrc
end

# make a single file of registration params
cat dfile.r*.1D > dfile_rall.1D

# compute motion magnitude time series: the Euclidean norm
# (sqrt(sum squares)) of the motion parameter derivatives
1d_tool.py -infile dfile_rall.1D -set_nruns 4 \
-derivative -collapse_cols euclidean_norm \
-write motion_${subj}_enorm.1D

# ----------------------------------------
# create the extents mask: mask_epi_extents+tlrc
# (this is a mask of voxels that have valid data at every TR)
3dMean -datum short -prefix rm.epi.mean rm.epi.min.r*.HEAD
3dcalc -a rm.epi.mean+tlrc -expr 'step(a-0.999)' -prefix mask_epi_extents

# and apply the extents mask to the EPI data
# (delete any time series with missing data)
foreach run ( $runs )
3dcalc -a rm.epi.nomask.r$run+tlrc -b mask_epi_extents+tlrc \
-expr 'a*b' -prefix pb02_${subj}_r${run}_volreg
end

# create an anat_final dataset, aligned with stats
3dcopy MPRage160sl7degp2SC01821_ns+tlrc anat_final.${subj}

# ================================== blur ==================================
# blur each volume of each run
foreach run ( $runs )
3dBlurToFWHM -FWHM 6 -mask mask_epi_extents+tlrc \
-input pb02_${subj}_r${run}_volreg+tlrc \
-prefix pb03_${subj}_r${run}_blur
end

# ================================== mask ==================================
# create 'full_mask' dataset (union mask)
foreach run ( $runs )
3dAutomask -dilate 1 -prefix rm.mask_r$run pb03_${subj}_r${run}_blur+tlrc
end

# get mean and compare it to 0 for taking 'union'
3dMean -datum short -prefix rm.mean rm.mask*.HEAD
3dcalc -a rm.mean+tlrc -expr 'ispositive(a-0)' -prefix full_mask_$subj

# ---- create subject anatomy mask, mask_anat.$subj+tlrc ----
# (resampled from tlrc anat)
3dresample -master full_mask_$subj+tlrc -input \
MPRage160sl7degp2SC01821_ns+tlrc \
-prefix rm.resam.anat

# convert to binary anat mask; fill gaps and holes
3dmask_tool -dilate_input 5 -5 -fill_holes -input rm.resam.anat+tlrc \
-prefix mask_anat.$subj

# compute overlaps between anat and EPI masks
3dABoverlap -no_automask full_mask_$subj+tlrc mask_anat.$subj+tlrc \
|& tee out.mask_overlap.txt

# ---- create group anatomy mask, mask_group+tlrc ----
# (resampled from tlrc base anat, MNI152_1mm_uni+tlrc)
3dresample -master full_mask_$subj+tlrc -prefix ./rm.resam.group \
-input ~/abin/MNI152_1mm_uni+tlrc

# convert to binary group mask; fill gaps and holes
3dmask_tool -dilate_input 5 -5 -fill_holes -input rm.resam.group+tlrc \
-prefix mask_group

# ================================= scale ==================================
# scale each voxel time series to have a mean of 100
# (be sure no negatives creep in)
# (subject to a range of [0,200])
foreach run ( $runs )
3dTstat -prefix rm.mean_r$run pb03_${subj}_r${run}_blur+tlrc
3dcalc -a pb03_${subj}_r${run}_blur+tlrc -b rm.mean_r$run+tlrc \
-c mask_epi_extents+tlrc \
-expr 'c * min(200, a/b*100)*step(a)*step(b)' \
-prefix pb04_${subj}_r${run}_scale
end

# ================================ QA =================================

# Already got outcount, motion, md, now getting 3dTqual and mean signal
# Then concatenating each into an allruns file
# Then ploting as jpeg and moving to QA folder
foreach run ( $runs )
#3dTqual: compute quality index for time series data
3dTqual -range pb04_${subj}_r${run}_scale+tlrc > \
3dTqual_r${run}.1D
1dplot -jpeg ${subj}_r${run}_3dTqual -one -title \
'Data quality across time' -xlabel 'Time in TRs' -ylabel \
'Signal variability and bands +-3.5SD' 3dTqual_r${run}.1D

#3dROIstats: calculate mean values in brain and then plot
3dROIstats -quiet -mask mask_epi_extents+tlrc \
pb04_${subj}_r${run}_scale+tlrc > meanscale_r${run}.1D
1dplot -nopush -jpg ${subj}_r${run}_psc -title \
'Mean Signal over Time' -xlabel 'Time in TRs' -ylabel \
'Mean signal level' meanscale_r${run}.1D

#plot Motion and MaxDisplacemnt and Outcount
1dplot -jpg ${subj}_r${run}_md -title \
'Maximum movement collapsed across planes' md_r${run}.1D
1dplot -volreg -jpg ${subj}_r${run}_motion motion_r${run}.1D
1dplot -jpg ${subj}_r${run}_oc -one -title \
'Histogram of outlying voxels' -xlabel 'Time in TRs' \
-ylabel 'Outlier count' outcount.r${run}.1D
# I'd prefer to use outcount_r$run.1D
end

#Plot graphs concatenated together
cat 3dTqual_r*.1D > 3dTqual_all.1D
cat meanscale_r*.1D > meanscale_all.1D
cat md_r*.1D > md_all.1D
cat motion_r*.1D > motion_all.1D
1dplot -jpeg ${subj}_all_3dTqual -one -title \
'Data quality across time' -xlabel 'Time in TRs' -ylabel \
'Signal variability and bands +-3.5SD' 3dTqual_all.1D
1dplot -nopush -jpg ${subj}_all_psc -title 'Mean Signal over Time' \
-xlabel 'Time in TRs' -ylabel 'Mean signal level' \
meanscale_all.1D
1dplot -jpg ${subj}_all_md -title \
'Maximum movement collapsed across planes' md_all.1D
1dplot -volreg -jpg ${subj}_all_motion motion_all.1D
1dplot -jpg ${subj}_all_oc -one -title \
'Histogram of outlying voxels' -xlabel 'Time in TRs' -ylabel \
'Outlier count' outcount_rall.1D

mkdir QA QA/individual_runs QA/all_runs
mv *_all*.jpg QA/all_runs
mv *_r?*.jpg QA/individual_runs
mv *all.1D QA/all_runs/
mv *r?.1D QA/individual_runs/


# ================== auto block: generate review scripts ===================

# generate a review script for the unprocessed EPI data
gen_epi_review.py -script @epi_review.$subj \
-dsets pb00_${subj}_r*_tcat+orig.HEAD

# generate scripts to review single subject results
# (try with defaults, but do not allow bad exit status)
gen_ss_review_scripts.py -exit0

# ========================== auto block: finalize ==========================

# remove temporary files
\rm -fr rm.* awpy

# move preprocessing files to 'preproc.data' directory
mkdir preproc.data
mv dfile.r*.1D pb*.$subj.r*.* outcount.* preproc.data

# if the basic subject review script is here, run it
# (want this to be the last text output)
if ( -e @ss_review_basic ) ./@ss_review_basic |& tee out.ss_review.$subj.txt

# return to parent directory
cd ..

echo "execution finished: `date`"




# ==========================================================================
# script generated by the command:
#
# afni_proc.py -script Single-Preprocessing.sh -scr_overwrite -out_dir \
# '${subj}' -blocks tshift align tlrc volreg blur mask scale -copy_anat \
# /Volumes/BuckleyProjects/RAWorganization/Imaging/MBMF/MPRage160sl7degp2SC01821.nii.gz \
# -dsets \
# /Volumes/BuckleyProjects/RAWorganization/Imaging/MBMF/BOLDRUN1SC01821A.nii.gz \
# /Volumes/BuckleyProjects/RAWorganization/Imaging/MBMF/BOLDRUN2SC01821A.nii.gz \
# /Volumes/BuckleyProjects/RAWorganization/Imaging/MBMF/BOLDRUN3SC01821A.nii.gz \
# /Volumes/BuckleyProjects/RAWorganization/Imaging/MBMF/BOLDRUN4SC01821A.nii.gz \
# -volreg_align_to first -volreg_interp -heptic -volreg_opts_vr \
# -maxdisp1D 'md_r${run}.1D' -1Dfile 'motion_r${run}}.1D' \
# -volreg_align_e2a -subj_curly -sep_char _ -blur_size 6 -align_opts_aea \
# -AddEdge -move_preproc_files -tlrc_NL_warp -tlrc_base \
# ~/abin/MNI152_1mm_uni+tlrc -blur_to_fwhm \
# -volreg_tlrc_warp