Hi, I am trying to run hcp_icafix on all the resting data in a study (2 pairs of reverse PE encoded scans), and my main goal is the concatenated output file (fMRI_CONCAT_ALL). The only functional scans in the study are resting scans, so I let the command run with the default settings (i.e., without specifying which bolds to use). Everything seems to run as expected for the first ~2hrs and then errors out. I combed through the error messages in the log files but it’s not clear what the exact problem is – any advice is appreciated, as always! Thanks so much.
Hi Jure, appreciate the quick reply! Thanks for bringing me up to speed, I will use the container on Grace from here on. I tried what you suggested (as written) and it errors out after a few minutes:
> (qunex) [login2 comlogs]$ cat error_hcp_icafix_fMRI_CONCAT_ALL_pb8993_2024-04-23_11.40.08.123007.log
> # Generated by QuNex 0.99.2 on 2024-04-23_11.40.08.123007#
> ------------------------------------------------------------
> Running external command via QuNex:
>
> /opt/HCP/HCPpipelines/ICAFIX/hcp_fix_multi_run --fmri-names="/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/1/1@/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/2/2@/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/3/3@/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/4/4" --high-pass=0 --concat-fmri-name="/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/fMRI_CONCAT_ALL/fMRI_CONCAT_ALL"
>
> Test file:
> /gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/fMRI_CONCAT_ALL/fMRI_CONCAT_ALL_hp0_clean.nii.gz
> ------------------------------------------------------------
>
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: arguments: --fmri-names=/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/1/1@/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/2/2@/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/3/3@/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/4/4 --high-pass=0 --concat-fmri-name=/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/fMRI_CONCAT_ALL/fMRI_CONCAT_ALL
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: fmrilist: /gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/1/1@/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/2/2@/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/3/3@/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/4/4
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: hp: 0
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: ConcatName: /gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/fMRI_CONCAT_ALL/fMRI_CONCAT_ALL
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: doMotionRegression: FALSE
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: TrainingData:
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: FixThresh: 10
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: DeleteIntermediates: FALSE
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: RewishartThresh: 0
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: volwisharts: 2
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: ciftiwisharts: 3
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: icadimmode: default
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: ProcessingMode: HCPStyleData
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: HCPPIPEDIR: /opt/HCP/HCPpipelines
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: CARET7DIR: /opt/workbench/workbench/bin_rh_linux64
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: FSLDIR: /opt/fsl/fsl
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: FSL_FIXDIR: /opt/fsl/fix
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: HCPCIFTIRWDIR: /opt/HCP/HCPpipelines/global/matlab/cifti-matlab
> Tue Apr 23 11:40:08 EDT 2024:hcp_fix_multi_run: Showing HCP Pipelines version
> Post-v4.7.0-5df95e4
> Tue Apr 23 11:40:11 EDT 2024:hcp_fix_multi_run: Showing Connectome Workbench (wb_command) version
> Connectome Workbench
> Type: Command Line Application
> Version: 1.5.0
> Qt Compiled Version: 5.7.0
> Qt Runtime Version: 5.7.0
> Commit: 76441605d20536d81034a64c9d34ec393aa528b9
> Commit Date: 2021-02-16 13:46:47 -0600
> Compiled with OpenMP: YES
> Compiler: g++ (/export/gcc-4.8.5/bin)
> Compiler Version: 4.8.5
> Compiled Debug: NO
> Operating System: Linux
> Tue Apr 23 11:40:11 EDT 2024:hcp_fix_multi_run: Showing FSL version
> Tue Apr 23 11:40:11 EDT 2024:hcp_fix_multi_run: INFO: Determined that the FSL version in use is 6.0.7.1
> Tue Apr 23 11:40:11 EDT 2024:hcp_fix_multi_run: FSL version: 6.0.7.1
> Tue Apr 23 11:40:11 EDT 2024:hcp_fix_multi_run: FIX version: 1.06.15
> Tue Apr 23 11:40:11 EDT 2024:hcp_fix_multi_run: FSL_FIX_MATLAB_MODE: 0
> Tue Apr 23 11:40:11 EDT 2024:hcp_fix_multi_run: hp=0 corresponds to a linear detrend
> Tue Apr 23 11:40:11 EDT 2024:hcp_fix_multi_run: PWD: /gpfs/gibbs/pi/n3/Studies/ExTx3
> Tue Apr 23 11:40:11 EDT 2024:hcp_fix_multi_run: Looping over files and doing highpass to each of them
> Tue Apr 23 11:40:11 EDT 2024:hcp_fix_multi_run: Top of loop through fmris: fmri: /gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/1/1
> parsed 'TCS - MEAN' as 'TCS - MEAN'
> Tue Apr 23 11:44:07 EDT 2024:hcp_fix_multi_run: tr: 0.800000
> Tue Apr 23 11:44:07 EDT 2024:hcp_fix_multi_run: processing FMRI file 1 with highpass 0
> Tue Apr 23 11:44:07 EDT 2024:hcp_fix_multi_run: MATLAB_COMPILER_RUNTIME: /opt/matlab/v93
> Tue Apr 23 11:44:07 EDT 2024:hcp_fix_multi_run: Run compiled MATLAB with command...
> Tue Apr 23 11:44:07 EDT 2024:hcp_fix_multi_run: /opt/fsl/fix/call_matlab.sh -c /opt/matlab/v93 -b /opt/HCP/HCPpipelines/ICAFIX/scripts/Compiled_functionhighpassandvariancenormalize -f functionhighpassandvariancenormalize 0.800000 0 1 /opt/workbench/workbench/bin_rh_linux64/wb_command 2 3 default
> ------------------------------------------
> Setting up environment variables
> ---
> LD_LIBRARY_PATH is .:/opt/matlab/v93/runtime/glnxa64:/opt/matlab/v93/bin/glnxa64:/opt/matlab/v93/sys/os/glnxa64:/opt/matlab/v93/sys/opengl/lib/glnxa64
> Warning: functionhighpassandvariancenormalize: hp=0 will be interpreted as polynomial detrending of order 1
> > In functionhighpassandvariancenormalize (line 86)
> Elapsed time is 2.482818 seconds.
> Warning: MATLAB was unable to open a pseudo-tty: Unknown error [0,1].
> The unix() and ! commands will not work in this MATLAB session. Other commands which depend upon unix() and ! will also fail. Your system may be running low on resources. If the problem persists after a reboot, check with your system administrator and confirm that your pty subsystem is properly configured.
> > In dos (line 66)
> In save_avw_hdr (line 28)
> In save_avw (line 26)
> In functionhighpassandvariancenormalize (line 121)
> Error using matlab.system.internal.executeCommand
> The unix() and ! commands are disabled.
>
> Error in dos (line 66)
>
>
> Error in save_avw_hdr (line 28)
>
>
> Error in save_avw (line 26)
>
>
> Error in functionhighpassandvariancenormalize (line 121)
>
> MATLAB:bang:CommandsDisabled
> Tue Apr 23 11:44:19 EDT 2024:hcp_fix_multi_run: While running '/opt/HCP/HCPpipelines/ICAFIX/hcp_fix_multi_run --fmri-names=/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/1/1@/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/2/2@/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/3/3@/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/4/4 --high-pass=0 --concat-fmri-name=/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/fMRI_CONCAT_ALL/fMRI_CONCAT_ALL':
> Tue Apr 23 11:44:19 EDT 2024:hcp_fix_multi_run: While running '/opt/HCP/HCPpipelines/ICAFIX/hcp_fix_multi_run --fmri-names=/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/1/1@/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/2/2@/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/3/3@/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/4/4 --high-pass=0 --concat-fmri-name=/gpfs/gibbs/pi/n3/Studies/ExTx3/sessions/pb8993/hcp/pb8993/MNINonLinear/Results/fMRI_CONCAT_ALL/fMRI_CONCAT_ALL':
> Tue Apr 23 11:44:19 EDT 2024:hcp_fix_multi_run: ERROR: '/opt/fsl/fix/call_matlab.sh -c /opt/matlab/v93 -b /opt/HCP/HCPpipelines/ICAFIX/scripts/Compiled_functionhighpassandvariancenormalize -f functionhighpassandvariancenormalize 0.800000 0 1 /opt/workbench/workbench/bin_rh_linux64/wb_command 2 3 default' command failed with return code: 255
> Tue Apr 23 11:44:19 EDT 2024:hcp_fix_multi_run: ERROR: '/opt/fsl/fix/call_matlab.sh -c /opt/matlab/v93 -b /opt/HCP/HCPpipelines/ICAFIX/scripts/Compiled_functionhighpassandvariancenormalize -f functionhighpassandvariancenormalize 0.800000 0 1 /opt/workbench/workbench/bin_rh_linux64/wb_command 2 3 default' command failed with return code: 255
>
> ===> ERROR: Command returned with nonzero exit code
> ---------------------------------------------------
> script: hcp_fix_multi_run
> stopped at line: 478
> call: "${matlab_cmd[@]}"
> expanded call: /opt/fsl/fix/call_matlab.sh -c /opt/matlab/v93 -b /opt/HCP/HCPpipelines/ICAFIX/scripts/Compiled_functionhighpassandvariancenormalize -f functionhighpassandvariancenormalize 0.800000 0 1 /opt/workbench/workbench/bin_rh_linux64/wb_command 2 3 default
> hostname: r805u25n01.grace.ycrc.yale.edu
> exit code: 255
> ---------------------------------------------------
>
> ===> Aborting execution!
This is a weird issue that we saw only when using the Grace cluster, not sure what is happening there. You could try using a different container for example qunex_suite-0.95.3a.sif.
Is there a way for me or for some of my Yale colleagues to get access to this study? Since this is super pressing, this might be the easiest way as I can take a look directly at what is happening.
Good news , I managed to successfully run icafix for one of the sessions. I think it might be the easiest if I run everything for all of them now, so I will just do that. You can keep track of the progress in /gpfs/gibbs/pi/n3/Studies/ExTx3/processing/logs/icafix_jd/comlogs.
The issue you are seeing started popping up after the Grace update and should be resolved once we release the next container (0.100.0).
hcp_icafix finished successfully for all sessions (see /gpfs/gibbs/pi/n3/Studies/ExTx3/processing/logs/icafix_jd/comlogs). Steps that follow (hcp_msmall …) should now work without any additional interventions.