Hi Paul,
thank you, this makes me rethink my workflow. I am also concerned whether this issue might affect the analysis itself - I'll get to it at the end.
Let me describe the workflow quickly - you may or may not remember it from the previous conversations, but I also want to make it clear in case anyone else reads it and learns from it.
So the pipeline is based on MACAQUE_REST_DEMO, and, like the demo, it consists of three stages 1. aligning the individual anatomy to the NMT template via animal_warper (AW), 2. preprocessing and regression via afni_proc (AP), 3. actual correlation stuff. However I restructured the scripts so that they can run in a far more parallelized way on HPC, beyond what OpenMP allows within some AFNI programs. So in 1/AW all monkeys are processed in parallel as separate jobs, in 2/AP it goes much further as each monkey, session and blurred/not-blurred analysis are processed in parallel. This is possible in HPC, while when running locally things run serially similar to the original demo. All of that is controlled automatically by my script.
Also I can run each stage separately by setting script variables (sort of similar to $1 of the demo script), and occasionally I would run only a subset (i.e., one stage, one monkey, one session) for testing purposes.
I have been mostly working on stage 2/AP recently, as I am struggling with getting anat-EPI alignment to work in some cases (Daniel about it, knows as he has been helping me a lot).
For this reason, what I would typically do is have the results of 1/AW unchanged and originally produced on HPC, and I would modify 2/AP and test-run it on a subset of data locally. This worked well, I am pretty sure also including QC, as this is the place I would normally look first for a quick look at the alignment. I did not realize there were any risks, as both environments have a copy of my data and a copy of the template, and the script is provided with the appropriate, environment-specific paths to both.
What I did yesterday was: I made changes to 1/AW and ran it locally to check a new idea. Results were bad, so I reverted the folder with the AW results to the original state. Then QC in 2/AP stopped working, as I described in the beginning of this thread. I realized though that "reverted" may have been incomplete ("some other things were messed up"), this is why I did not go into details of that, but I copied older 1/AW results from HPC to revert it for sure, and re-run 2/AP locally on a subset overnight. And I got no QC again.
So I will look into the environment variables, until now I was not aware that some information is passed outside the directories where AW and AP place their results, and/or through these directories but in a hidden way like file histories.
But now I have this question whether the issue of the template location may affect the analysis itself. Because I have been puzzled for quite a while by inconsistencies between HPC and local results.
I test-ran 2/AP with a new set of parameters on a subset of data locally, but using 1/AW results previously calculated on HPC. The EPI-anat alignment would look OK (if not perfect), so I would run 2/AP on the entire dataset on HPC. The alignment for the same session would be completely off - which looked as if the procedure was sensitive to the computer or OS it is running on. Could it be that it was another manifestation of the template location issue?
Edited 1 time(s). Last edit at 04/22/2021 10:28AM by Pawel.
