[RESOLVED] Issues with hcp_icafix

helen.pushkarskaya · May 31, 2024, 9:47pm

I have difficulties running hcp_icafix. Looks like that I maybe don’
t have access to some functions?

Unable to find MATLAB Compiler Runtime
/gpfs/gibbs/pi/n3/software/fsl/fix/call_matlab.sh

Is it a problem with my account?

My call is:

qunex hcp_icafix \
    --batchfile=/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/processing/batch_hcp_04012024_pb1047.txt\
    --sessionsfolder=/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions \
    --hcp_icafix_bolds="1,2,3,4,5,6,7" \
    --scheduler="SLURM,time=1-00:00:00,ntasks=1,cpus-per-task=1,mem-per-cpu=40000,partition=day"

The error log is: /gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/processing/logs/comlogs/error_hcp_icafix_1_pb1047_2024-05-30_00.15.27.192199.log

Thank you in advance.

helen.pushkarskaya · June 1, 2024, 2:02am

I tried running

export LD_LIBRARY_PATH="${LD_LIBRARY_PATH:+${LD_LIBRARY_PATH}:}\
> /vast/palmer/apps/avx2/software/MATLAB/2023a/runtime/glnxa64:\
> /vast/palmer/apps/avx2/software/MATLAB/2023a/bin/glnxa64:\
> /vast/palmer/apps/avx2/software/MATLAB/2023a/sys/os/glnxa64:\
> /vast/palmer/apps/avx2/software/MATLAB/2023a/extern/bin/glnxa64"

But it did not help. Same error.

demsarjure · June 1, 2024, 6:19am

Helen,

it seem you are trying to run QuNex outside of the container, this is not advised. Please always use the container:

qunex_container hcp_icafix \
    --batchfile=/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/processing/batch_hcp_04012024_pb1047.txt\
    --sessionsfolder=/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions \
    --hcp_icafix_bolds="1,2,3,4,5,6,7" \
    --container="<provide the path to the latest container (0.99.2d right now) here>" \
    --scheduler="SLURM,time=1-00:00:00,ntasks=1,cpus-per-task=1,mem-per-cpu=40000,partition=day"

helen.pushkarskaya · June 1, 2024, 3:32pm

Thank you. Will make sure in the future.
Trying now.

helen.pushkarskaya · June 1, 2024, 4:57pm

Hi Jure,
I tried several version of qunex container, and was able to move further. Unfortunately, still getting an error:

 extract features
FIX Classifying components in Melodic directory: 1_hp2000.ica using training file: /opt/fsl/fix/training_files/HCP_hp2000.RData and threshold 10
FIX features not already generated, so creating them now.
FIX Feature extraction for Melodic output directory: 1_hp2000.ica
 create edge masks
 run FAST
 registration of standard space masks
 extract features
No valid labelling file specified
Sat Jun  1 12:51:39 EDT 2024:hcp_fix: Done running FIX
Could not find a supported file with prefix "1_hp2000.ica/filtered_func_data_clean"
Sat Jun  1 12:51:39 EDT 2024:hcp_fix: While running '/opt/HCP/HCPpipelines/ICAFIX/hcp_fix /gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions/pb1047/hcp/pb1047/MNINonLinear/Results/1/1 2000 TRUE HCP_hp2000.RData 10 False':
Sat Jun  1 12:51:39 EDT 2024:hcp_fix: ERROR: '/opt/fsl/fsl/bin/immv' command failed with return code: 1
Sat Jun  1 12:51:39 EDT 2024:hcp_fix: ERROR: '/opt/fsl/fsl/bin/immv' command failed with return code: 1

===> ERROR: Command returned with nonzero exit code
---------------------------------------------------
         script: hcp_fix
stopped at line: 462
           call: $FSLDIR/bin/immv ${fmrihp}.ica/filtered_func_data_clean ${fmrihp}_clean
  expanded call: /opt/fsl/fsl/bin/immv 1_hp2000.ica/filtered_func_data_clean 1_hp2000_clean
       hostname: r906u41n03.grace.ycrc.yale.edu
      exit code: 1
---------------------------------------------------

===> Aborting execution!

I can see that Rachael had some difficulties with a similar call, and you were able to resolve it. Do I need to wait for the latest container?

demsarjure · June 2, 2024, 12:35pm

hcp_icafix works fine on pretty much all systems with 0.99.2d, a bunch of other users as well as ourselves are running it without any issues. Can you upload full logs here?

Also, you are resorting to the old “single fix” that was deprecated by HCP folks (you are doing this by explicitly setting --hcp_icafix_bolds="1,2,3,4,5,6,7"), why are you setting this and not leaving it blank so QuNex uses the defaults (which is to perform fix over all BOLDs)? To use “multi fix” which is the traditional way to go, you need to leave the parameter empty, of if you want to use only a subset of bolds set --hcp_icafix_bolds="fMRI_CONCAT_ALL=1,2,3,4,5,6,7". Replace fMRI_CONCAT_ALL with whatever you want the name to be, for example fMRI_CONCAT_REST if you are concatenating REST BOLDs only. See hcp_icafix — QuNex documentation for additional details.

Best, Jure

helen.pushkarskaya · June 2, 2024, 3:13pm

Thank you so much!
Trying it no.

helen.pushkarskaya · June 2, 2024, 4:21pm

Unfortunately, still struggling.

Tried different versions of the call:

qunex_container hcp_icafix \
    --batchfile=/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/processing/batch_hcp_04012024_pb1047.txt\
    --sessionsfolder=/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions \
    --hcp_icafix_bolds="run1:1|run2:2|run3:3|run4:4|run5:5|run6:6|run7:7" \
    --overwrite="yes"  \
    --logfolder="/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/processing/logs/icafix" \
    --container="/gpfs/gibbs/pi/n3/software/Singularity/qunex_suite-0.99.2d.sif" \
    --scheduler="SLURM,time=1-00:00:00,ntasks=1,cpus-per-task=1,--job-name=HP_icafix,mem-per-cpu=60000,partition=day"

qunex_container hcp_icafix \
    --batchfile=/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/processing/batch_hcp_04012024_pb1047.txt\
    --sessionsfolder=/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions \
    --hcp_icafix_bolds="try1:1,2,3,4,5,6,7" \
    --overwrite="yes"  \
    --parelements="4"  \
    --logfolder="/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/processing/logs/icafix" \
    --container="/gpfs/gibbs/pi/n3/software/Singularity/qunex_suite-0.99.2d.sif" \
    --scheduler="SLURM,time=1-00:00:00,ntasks=4,cpus-per-task=1,--job-name=HP_icafix,mem-per-cpu=40000,partition=day"

qunex_container hcp_icafix \
    --batchfile=/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/processing/batch_hcp_04012024_pb1047.txt\
    --sessionsfolder=/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions \
    --parelements="4"  \
    --overwrite="yes"  \
    --logfolder="/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/processing/logs/icafix" \
    --container="/gpfs/gibbs/pi/n3/software/Singularity/qunex_suite-0.99.2d.sif" \
    --scheduler="SLURM,time=1-00:00:00,ntasks=4,cpus-per-task=1,--job-name=HP_icafix,mem-per-cpu=40000,partition=day"

Getting the same error:

...Sun Jun  2 12:15:39 EDT 2024:hcp_fix_multi_run: /opt/fsl/fix/call_matlab.sh -c /opt/matlab/v93 -b /opt/HCP/HCPpipelines/ICAFIX/scripts/Compiled_functionhighpassandvariancenormalize -f functionhighpassandvariancenormalize 0.800000 0 1 /opt/workbench/workbench/bin_rh_linux64/wb_command  2 3 default
------------------------------------------
Setting up environment variables
---
LD_LIBRARY_PATH is .:/opt/matlab/v93/runtime/glnxa64:/opt/matlab/v93/bin/glnxa64:/opt/matlab/v93/sys/os/glnxa64:/opt/matlab/v93/sys/opengl/lib/glnxa64
Warning: functionhighpassandvariancenormalize: hp=0 will be interpreted as polynomial detrending of order 1
> In functionhighpassandvariancenormalize (line 86)
Elapsed time is 1.881200 seconds.
Warning: MATLAB was unable to open a pseudo-tty: Unknown error [0,1].
The unix() and ! commands will not work in this MATLAB session.  Other commands which depend upon unix() and ! will also fail.  Your system may be running low on resources.  If the problem persists after a reboot, check with your system administrator and confirm that your pty subsystem is properly configured.
> In dos (line 66)
  In save_avw_hdr (line 28)
  In save_avw (line 26)
  In functionhighpassandvariancenormalize (line 121)
Error using matlab.system.internal.executeCommand
The unix() and ! commands are disabled.

Error in dos (line 66)
Error in save_avw_hdr (line 28)
Error in save_avw (line 26)
Error in functionhighpassandvariancenormalize (line 121)

MATLAB:bang:CommandsDisabled
Sun Jun  2 12:15:50 EDT 2024:hcp_fix_multi_run: While running '/opt/HCP/HCPpipelines/ICAFIX/hcp_fix_multi_run --fmri-names=/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions/pb1047/hcp/pb1047/MNINonLinear/Results/1/1@/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions/pb1047/hcp/pb1047/MNINonLinear/Results/2/2@/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions/pb1047/hcp/pb1047/MNINonLinear/Results/3/3@/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions/pb1047/hcp/pb1047/MNINonLinear/Results/4/4@/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions/pb1047/hcp/pb1047/MNINonLinear/Results/5/5@/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions/pb1047/hcp/pb1047/MNINonLinear/Results/6/6@/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions/pb1047/hcp/pb1047/MNINonLinear/Results/7/7 --high-pass=0 --concat-fmri-name=/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions/pb1047/hcp/pb1047/MNINonLinear/Results/try1/try1':
Sun Jun  2 12:15:50 EDT 2024:hcp_fix_multi_run: While running '/opt/HCP/HCPpipelines/ICAFIX/hcp_fix_multi_run --fmri-names=/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions/pb1047/hcp/pb1047/MNINonLinear/Results/1/1@/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions/pb1047/hcp/pb1047/MNINonLinear/Results/2/2@/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions/pb1047/hcp/pb1047/MNINonLinear/Results/3/3@/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions/pb1047/hcp/pb1047/MNINonLinear/Results/4/4@/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions/pb1047/hcp/pb1047/MNINonLinear/Results/5/5@/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions/pb1047/hcp/pb1047/MNINonLinear/Results/6/6@/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions/pb1047/hcp/pb1047/MNINonLinear/Results/7/7 --high-pass=0 --concat-fmri-name=/gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/sessions/pb1047/hcp/pb1047/MNINonLinear/Results/try1/try1':
Sun Jun  2 12:15:50 EDT 2024:hcp_fix_multi_run: ERROR: '/opt/fsl/fix/call_matlab.sh -c /opt/matlab/v93 -b /opt/HCP/HCPpipelines/ICAFIX/scripts/Compiled_functionhighpassandvariancenormalize -f functionhighpassandvariancenormalize 0.800000 0 1 /opt/workbench/workbench/bin_rh_linux64/wb_command  2 3 default' command failed with return code: 255
Sun Jun  2 12:15:50 EDT 2024:hcp_fix_multi_run: ERROR: '/opt/fsl/fix/call_matlab.sh -c /opt/matlab/v93 -b /opt/HCP/HCPpipelines/ICAFIX/scripts/Compiled_functionhighpassandvariancenormalize -f functionhighpassandvariancenormalize 0.800000 0 1 /opt/workbench/workbench/bin_rh_linux64/wb_command  2 3 default' command failed with return code: 255

===> ERROR: Command returned with nonzero exit code
---------------------------------------------------
         script: hcp_fix_multi_run
stopped at line: 478
           call: "${matlab_cmd[@]}"
  expanded call: /opt/fsl/fix/call_matlab.sh -c /opt/matlab/v93 -b /opt/HCP/HCPpipelines/ICAFIX/scripts/Compiled_functionhighpassandvariancenormalize -f functionhighpassandvariancenormalize 0.800000 0 1 /opt/workbench/workbench/bin_rh_linux64/wb_command  2 3 default
       hostname: r904u07n02.grace.ycrc.yale.edu
      exit code: 255
---------------------------------------------------
===> Aborting execution!

logs: /gpfs/gibbs/pi/n3/Studies/Pushkarskaya.RA/processing/logs/icafix

helen.pushkarskaya · June 2, 2024, 8:01pm

I switch a Matlab version to

--hcp_matlab_mode="octave" \

It noted an error in the log:

aerror: pkg: cannot create file 
aerror: called from
a    pkg at line 510 column 13
a    /home/hp245/.octaverc at line 5 column 1

but still seems to go further in the process. Fingers crossed

demsarjure · June 3, 2024, 6:18am

Yes, Octave might work. I cannot help you regarding MATLAB not working:

Warning: MATLAB was unable to open a pseudo-tty: Unknown error [0,1].
The unix() and ! commands will not work in this MATLAB session.  Other commands which depend upon unix() and ! will also fail.  Your system may be running low on resources.  If the problem persists after a reboot, check with your system administrator and confirm that your pty subsystem is properly configured.

This is a system issue. You should contact your system administrators.

Regarding the commands, the last one makes the most sense (at a glance), I do not know the details of your dataset. You can omit parelements here as icafix does not use it in that way.

Best, Jure

helen.pushkarskaya · June 3, 2024, 3:52pm

Thank you! Will do.
The code got to the end successfully.