I think you are missing “\” at the end of the lines with --processed_data and --bind.
Thanks for pointing that out. Unfortunately, even after including the "" I’m still getting the same errors.
Like Andraz said, there are some issues in your bash code as QuNex is not receiving the container parameter. In the last message, you said you included the "", you should be adding the \. So, the qunex_container block should read:
qunex_container import_hcp \
--sessionsfolder="${STUDY}/sessions" \
--sessions="${SESSIONS}" \
--inbox="${RAW_DATA}" \
--processed_data="${WORK_DIR}/preprocessed/${SESSIONS}" \
--archive="leave" \
--bind="${WORK_DIR}:${WORK_DIR}" \
--container="${QUNEX_CONTAINER}"
Hi,
Sorry for missing the " \ " in my previous post, although it was there in the bash script. I realized while typing this message, that not giving spaces between " " sign leads to the slash symbol vanish!
Here is the whole bash script. This is for testing, so the ‘for’ loop has only 2 subjects. In order to make sure that every command runs smoothly, I was including each command and testing the loop. But the moment I try to put everything together, it started throwing errors that certain files, at every step, are missing. Hence, I tried commenting out rest of the qunex commands except for one and check if its running smoothly. For example, if I have to run ‘hcp_fmri_surface’, then I’m commenting out the rest, making sure that the previous ones have already run and necessary files are there in the folders. Then just to test whether the rest of the script runs properly or not, I ran all the commands following ‘hcp_fmri_surface’, but the code stopped, and threw the error that it cannot find some file for hcp_icafix. I checked if that particular file exists in the said folder or not and it was already there. I have included the error logs for hcp_icafix as an example. This has been going on for every command so far. I basically want to run this bash script for every batch of 10 subjects.
Thanks,
Mona
#!/bin/bash
# This script reads each entry from the HCP_EP subject list and runs the qunex commands to minimally process dataset.
cd /sc/arion/projects/WenglerLab_SczEnsembles/qunex
#the file to read from
file="HCPepSubjects.txt";
#Total number of subjects (lines) in the dataset
line_count=$(wc -l < HCPepSubjects.txt);
echo $line_count
#parameter prep:
ml restore myenv
export WORK_DIR="/sc/arion/projects/WenglerLab_SczEnsembles"
export STUDY="$WORK_DIR/qunex/HCP_EP"
export INPUT_PARAM_FILE="$WORK_DIR/qunex/params_temp.txt"
export RUN_TURNKEY_STEPS="hcp_pre_freesurfer,hcp_freesurfer,hcp_post_freesurfer,hcp_fmri_volume,hcp_fmri_surface"
start_line=3
end_line=4
# Initialize a line counter
line_num=0
# Loop through the file and read line by line
while IFS= read -r line
do
((line_num++)) # Increment line number
# Skip lines until start_line is reached
if (( line_num < start_line )); then
continue
fi
# Stop when we go past end_line
if (( line_num > end_line )); then
break
fi
#Starting qunex pipeline
export SESSIONS="$line"
echo $SESSIONS
qunex_container import_hcp \
--sessionsfolder="$STUDY/sessions" \
--sessions="$SESSIONS" \
--inbox="$RAW_DATA" \
--processed_data="$WORK_DIR/preprocessed/$SESSIONS" \
--archive="leave" \
--bind="$WORK_DIR:$WORK_DIR" \
--container="$QUNEX_CONTAINER"
qunex_container setup_hcp \
--sessionsfolder="$STUDY/sessions" \
--sessions="$SESSIONS" \
--hcp_filename='userdefined' \
--bind="$WORK_DIR:$WORK_DIR" \
--container="$QUNEX_CONTAINER"
qunex_container create_batch \
--sessionsfolder="$STUDY/sessions" \
--sessions="$SESSIONS" \
--targetfile="$STUDY/processing/batch.txt" \
--overwrite='append' \
--paramfile="$INPUT_PARAM_FILE" \
--container="$QUNEX_CONTAINER"
qunex_container hcp_pre_freesurfer \
--sessionsfolder="${STUDY}/sessions" \
--batchfile="${STUDY}/processing/batch.txt" \
--hcp_filename='userdefined' \
--container="$QUNEX_CONTAINER"
qunex_container hcp_freesurfer \
--sessionsfolder="${STUDY}/sessions" \
--batchfile="${STUDY}/processing/batch.txt" \
--hcp_filename='userdefined' \
--container="$QUNEX_CONTAINER"
qunex_container hcp_post_freesurfer \
--sessionsfolder="${STUDY}/sessions" \
--batchfile="${STUDY}/processing/batch.txt" \
--hcp_filename='userdefined' \
--container="$QUNEX_CONTAINER"
qunex_container run_qc \
--sessionsfolder="${STUDY}/sessions" \
--batchfile="${STUDY}/processing/batch.txt" \
--sessions="$SESSIONS" \
--modality='T1w' \
--container="$QUNEX_CONTAINER"
qunex_container run_qc \
--sessionsfolder="${STUDY}/sessions" \
--batchfile="${STUDY}/processing/batch.txt" \
--sessions="$SESSIONS" \
--modality='T2w' \
--container="$QUNEX_CONTAINER"
qunex_container run_qc \
--sessionsfolder="${STUDY}/sessions" \
--batchfile="${STUDY}/processing/batch.txt" \
--sessions="$SESSIONS" \
--modality='myelin' \
--overwrite='yes' \
--container="$QUNEX_CONTAINER"
qunex_container hcp_fmri_volume \
--sessionsfolder="${STUDY}/sessions" \
--batchfile="${STUDY}/processing/batch.txt" \
--hcp_filename='userdefined' \
--parelements="4" \
--container="$QUNEX_CONTAINER"
qunex_container hcp_fmri_surface \
--sessionsfolder="${STUDY}/sessions" \
--batchfile="${STUDY}/processing/batch.txt" \
--hcp_filename='userdefined' \
--parelements="4" \
--container="$QUNEX_CONTAINER"
qunex_container hcp_icafix \
--sessionsfolder="${STUDY}/sessions" \
--batchfile="${STUDY}/processing/batch.txt" \
--bind="$WORK_DIR:$WORK_DIR" \
--hcp_matlab_mode='octave' \
--container="$QUNEX_CONTAINER"
qunex_container hcp_post_fix \
--sessionsfolder="${STUDY}/sessions" \
--batchfile="${STUDY}/processing/batch.txt" \
--hcp_matlab_mode='octave' \
--container="$QUNEX_CONTAINER"
qunex_container hcp_msmall \
--sessionsfolder="${STUDY}/sessions" \
--batchfile="${STUDY}/processing/batch.txt" \
--hcp_matlab_mode='octave' \
--container="$QUNEX_CONTAINER"
qunex_container hcp_dedrift_and_resample \
--sessionsfolder="${STUDY}/sessions" \
--batchfile="${STUDY}/processing/batch.txt" \
--hcp_matlab_mode='octave' \
--container="$QUNEX_CONTAINER"
qunex_container run_qc \
--sessionsfolder="${STUDY}/sessions" \
--batchfile="${STUDY}/processing/batch.txt" \
--sessions="$SESSIONS" \
--hcp_filename='userdefined' \
--modality='BOLD' \
--boldsuffix='Atlas_MSMAll'
--overwrite='yes' \
--container="$QUNEX_CONTAINER"
done < "$file"
Error logs:
# Generated by QuNex 0.100.0 [QX IO] on 2024-09-22_11.40.55.262035#
------------------------------------------------------------
Running external command via QuNex:
/opt/HCP/HCPpipelines/ICAFIX/hcp_fix_multi_run --fmri-names="/sc/arion/projects/WenglerLab_SczEnsembles/qunex/HCP_EP/sessions/1003_01_MR/hcp/1003_01_MR/MNINonLinear/Results/rfMRI_REST1_AP/rfMRI_REST1_AP@/sc/arion/projects/WenglerLab_SczEnsembles/qunex/HC$
Test file:
/sc/arion/projects/WenglerLab_SczEnsembles/qunex/HCP_EP/sessions/1003_01_MR/hcp/1003_01_MR/MNINonLinear/Results/fMRI_CONCAT_ALL/fMRI_CONCAT_ALL_hp0_clean.nii.gz
------------------------------------------------------------
Sun Sep 22 11:40:55 EDT 2024:hcp_fix_multi_run: arguments: --fmri-names=/sc/arion/projects/WenglerLab_SczEnsembles/qunex/HCP_EP/sessions/1003_01_MR/hcp/1003_01_MR/MNINonLinear/Results/rfMRI_REST1_AP/rfMRI_REST1_AP@/sc/arion/projects/WenglerLab_SczEnsembles/qunex/HCP_EP/$
Sun Sep 22 11:40:55 EDT 2024:hcp_fix_multi_run: fmrilist: /sc/arion/projects/WenglerLab_SczEnsembles/qunex/HCP_EP/sessions/1003_01_MR/hcp/1003_01_MR/MNINonLinear/Results/rfMRI_REST1_AP/rfMRI_REST1_AP@/sc/arion/projects/WenglerLab_SczEnsembles/qunex/HCP_EP/sessions/1003_$
Sun Sep 22 11:40:55 EDT 2024:hcp_fix_multi_run: hp: 0
Sun Sep 22 11:40:55 EDT 2024:hcp_fix_multi_run: ConcatName: /sc/arion/projects/WenglerLab_SczEnsembles/qunex/HCP_EP/sessions/1003_01_MR/hcp/1003_01_MR/MNINonLinear/Results/fMRI_CONCAT_ALL/fMRI_CONCAT_ALL
Sun Sep 22 11:40:55 EDT 2024:hcp_fix_multi_run: doMotionRegression: FALSE
Sun Sep 22 11:40:56 EDT 2024:hcp_fix_multi_run: TrainingData:
Sun Sep 22 11:40:56 EDT 2024:hcp_fix_multi_run: FixThresh: 10
Sun Sep 22 11:40:56 EDT 2024:hcp_fix_multi_run: DeleteIntermediates: FALSE
Sun Sep 22 11:40:56 EDT 2024:hcp_fix_multi_run: RewishartThresh: 0
Sun Sep 22 11:40:56 EDT 2024:hcp_fix_multi_run: volwisharts: 2
Sun Sep 22 11:40:56 EDT 2024:hcp_fix_multi_run: ciftiwisharts: 3
Sun Sep 22 11:40:56 EDT 2024:hcp_fix_multi_run: icadimmode: default
Sun Sep 22 11:40:56 EDT 2024:hcp_fix_multi_run: ProcessingMode: HCPStyleData
Sun Sep 22 11:40:56 EDT 2024:hcp_fix_multi_run: ConcatOnlyStr: FALSE
Sun Sep 22 11:40:56 EDT 2024:hcp_fix_multi_run: HCPPIPEDIR: /opt/HCP/HCPpipelines
Sun Sep 22 11:40:56 EDT 2024:hcp_fix_multi_run: CARET7DIR: /opt/workbench/workbench/bin_linux64
Sun Sep 22 11:40:56 EDT 2024:hcp_fix_multi_run: FSLDIR: /opt/fsl/fsl
Sun Sep 22 11:40:56 EDT 2024:hcp_fix_multi_run: FSL_FIXDIR: /opt/fsl/fix
Sun Sep 22 11:40:56 EDT 2024:hcp_fix_multi_run: HCPCIFTIRWDIR: /opt/HCP/HCPpipelines/global/matlab/cifti-matlab
Sun Sep 22 11:40:57 EDT 2024:hcp_fix_multi_run: Showing HCP Pipelines version
v4.8.0-rc-62c6fff
Sun Sep 22 11:41:58 EDT 2024:hcp_fix_multi_run: Showing Connectome Workbench (wb_command) version
Connectome Workbench
Type: Command Line Application
Version: 1.5.0
Qt Compiled Version: 5.7.0
Qt Runtime Version: 5.7.0
Commit: 76441605d20536d81034a64c9d34ec393aa528b9
Commit Date: 2021-02-16 13:46:47 -0600
Compiled with OpenMP: YES
Compiler: g++ (/home/caret/gcc/install/gcc-4.8.5/bin)
Compiler Version: 4.8.5
Compiled Debug: NO
Operating System: Linux
Sun Sep 22 11:42:00 EDT 2024:hcp_fix_multi_run: Showing FSL version
Sun Sep 22 11:42:00 EDT 2024:hcp_fix_multi_run: INFO: Determined that the FSL version in use is 6.0.7.11
Sun Sep 22 11:42:00 EDT 2024:hcp_fix_multi_run: FSL version: 6.0.7.11
Sun Sep 22 11:42:00 EDT 2024:hcp_fix_multi_run: FIX version: 1.06.15
Sun Sep 22 11:42:00 EDT 2024:hcp_fix_multi_run: FSL_FIX_MATLAB_MODE: 2
Sun Sep 22 11:42:00 EDT 2024:hcp_fix_multi_run: hp=0 corresponds to a linear detrend
Sun Sep 22 11:42:00 EDT 2024:hcp_fix_multi_run: PWD: /sc/arion/projects/WenglerLab_SczEnsembles/qunex
Sun Sep 22 11:42:00 EDT 2024:hcp_fix_multi_run: Looping over files and doing highpass to each of them
Sun Sep 22 11:42:00 EDT 2024:hcp_fix_multi_run: Top of loop through fmris: fmri: /sc/arion/projects/WenglerLab_SczEnsembles/qunex/HCP_EP/sessions/1003_01_MR/hcp/1003_01_MR/MNINonLinear/Results/rfMRI_REST1_AP/rfMRI_REST1_AP
While running:
/opt/workbench/workbench-1.5.0/bin_linux64/../exe_linux64/wb_command -cifti-reduce rfMRI_REST1_AP_Atlas.dtseries.nii MEAN rfMRI_REST1_AP_Atlas_mean.dscalar.nii
ERROR: failed to open file '/sc/arion/projects/WenglerLab_SczEnsembles/qunex/HCP_EP/sessions/1003_01_MR/hcp/1003_01_MR/MNINonLinear/Results/rfMRI_REST1_AP/rfMRI_REST1_AP_Atlas.dtseries.nii', file does not exist, or folder permissions prevent seeing it
Sun Sep 22 12:03:12 EDT 2024:hcp_fix_multi_run: While running '/opt/HCP/HCPpipelines/ICAFIX/hcp_fix_multi_run --fmri-names=/sc/arion/projects/WenglerLab_SczEnsembles/qunex/HCP_EP/sessions/1003_01_MR/hcp/1003_01_MR/MNINonLinear/Results/rfMRI_REST1_AP/rfMRI_REST1_AP@/sc/a$
Sun Sep 22 12:03:12 EDT 2024:hcp_fix_multi_run: While running '/opt/HCP/HCPpipelines/ICAFIX/hcp_fix_multi_run --fmri-names=/sc/arion/projects/WenglerLab_SczEnsembles/qunex/HCP_EP/sessions/1003_01_MR/hcp/1003_01_MR/MNINonLinear/Results/rfMRI_REST1_AP/rfMRI_REST1_AP@/sc/a$
Sun Sep 22 12:03:12 EDT 2024:hcp_fix_multi_run: ERROR: '/opt/workbench/workbench/bin_linux64/wb_command' command failed with return code: 255
Sun Sep 22 12:03:12 EDT 2024:hcp_fix_multi_run: ERROR: '/opt/workbench/workbench/bin_linux64/wb_command' command failed with return code: 255
===> ERROR: Command returned with nonzero exit code
---------------------------------------------------
script: hcp_fix_multi_run
While running:
/opt/workbench/workbench-1.5.0/bin_linux64/../exe_linux64/wb_command -cifti-reduce rfMRI_REST1_AP_Atlas.dtseries.nii MEAN rfMRI_REST1_AP_Atlas_mean.dscalar.nii
ERROR: failed to open file '/sc/arion/projects/WenglerLab_SczEnsembles/qunex/HCP_EP/sessions/1003_01_MR/hcp/1003_01_MR/MNINonLinear/Results/rfMRI_REST1_AP/rfMRI_REST1_AP_Atlas.dtseries.nii', file does not exist, or folder permissions prevent seeing it
Sun Sep 22 12:03:12 EDT 2024:hcp_fix_multi_run: While running '/opt/HCP/HCPpipelines/ICAFIX/hcp_fix_multi_run --fmri-names=/sc/arion/projects/WenglerLab_SczEnsembles/qunex/HCP_EP/sessions/1003_01_MR/hcp/1003_01_MR/MNINonLinear/Results/rfMRI_REST1_AP/rfMRI_REST1_AP@/sc/arion/projects/WenglerLab_S$
Sun Sep 22 12:03:12 EDT 2024:hcp_fix_multi_run: While running '/opt/HCP/HCPpipelines/ICAFIX/hcp_fix_multi_run --fmri-names=/sc/arion/projects/WenglerLab_SczEnsembles/qunex/HCP_EP/sessions/1003_01_MR/hcp/1003_01_MR/MNINonLinear/Results/rfMRI_REST1_AP/rfMRI_REST1_AP@/sc/arion/projects/WenglerLab_S$
Sun Sep 22 12:03:12 EDT 2024:hcp_fix_multi_run: ERROR: '/opt/workbench/workbench/bin_linux64/wb_command' command failed with return code: 255
Sun Sep 22 12:03:12 EDT 2024:hcp_fix_multi_run: ERROR: '/opt/workbench/workbench/bin_linux64/wb_command' command failed with return code: 255
===> ERROR: Command returned with nonzero exit code
---------------------------------------------------
script: hcp_fix_multi_run
stopped at line: 449
call: ${Caret7_Command} -cifti-reduce ${fmri}_Atlas.dtseries.nii MEAN ${fmri}_Atlas_mean.dscalar.nii
expanded call: /opt/workbench/workbench/bin_linux64/wb_command -cifti-reduce rfMRI_REST1_AP_Atlas.dtseries.nii MEAN rfMRI_REST1_AP_Atlas_mean.dscalar.nii
hostname: lc01g22
exit code: 255
---------------------------------------------------
===> Aborting execution!
The file list in the rfMRI_REST1_AP sub-folder is given below. The path to this sub-folder is:
/sc/arion/projects/WenglerLab_SczEnsembles/qunex/HCP_EP/sessions/1003_01_MR/hcp/1003_01_MR/MNINonLinear/Results/rfMRI_REST1_AP
.
├── brainmask_fs.2.nii.gz
├── fMRIQC
│ ├── 1003_01_MR_rfMRI_REST1_AP.fMRIQC.wb_scene
│ └── snapshots
│ ├── 1003_01_MR_rfMRI_REST1_AP.fMRIQC.wb_scene1.png
│ └── 1003_01_MR_rfMRI_REST1_AP.fMRIQC.wb_scene2.png
├── Movement_AbsoluteRMS_mean.txt
├── Movement_AbsoluteRMS.txt
├── Movement_Regressors_demean.txt
├── Movement_Regressors_dt.txt
├── Movement_Regressors.txt
├── Movement_RelativeRMS_mean.txt
├── Movement_RelativeRMS.txt
├── rfMRI_REST1_AP_Atlas.dtseries.nii
├── rfMRI_REST1_AP_Atlas_mean.dscalar.nii
├── rfMRI_REST1_AP_Atlas_nonzero.stats.txt
├── rfMRI_REST1_AP_demean.nii.gz
├── rfMRI_REST1_AP_dropouts.nii.gz
├── rfMRI_REST1_AP_finalmask.nii.gz
├── rfMRI_REST1_AP_finalmask.stats.txt
├── rfMRI_REST1_AP_fovmask.nii.gz
├── rfMRI_REST1_AP_hp0.ica
│ └── mc
│ └── prefiltered_func_data_mcf.par
├── rfMRI_REST1_AP_Jacobian.nii.gz
├── rfMRI_REST1_AP.L.native.func.gii
├── rfMRI_REST1_AP_mean.nii.gz
├── rfMRI_REST1_AP.nii.gz
├── rfMRI_REST1_AP_PhaseOne_gdc_dc.nii.gz
├── rfMRI_REST1_AP_PhaseTwo_gdc_dc.nii.gz
├── rfMRI_REST1_AP_pseudo_transmit_field.nii.gz
├── rfMRI_REST1_AP_pseudo_transmit_raw.nii.gz
├── rfMRI_REST1_AP.R.atlasroi.32k_fs_LR.func.gii
├── rfMRI_REST1_AP.R.native.func.gii
├── rfMRI_REST1_AP_SBRef.nii.gz
├── rfMRI_REST1_AP_SBRef_nomask.nii.gz
├── rfMRI_REST1_AP_sebased_bias.nii.gz
├── rfMRI_REST1_AP_sebased_bias.nii.gz_dilated.nii.gz
├── rfMRI_REST1_AP_sebased_reference.nii.gz
└── RibbonVolumeToSurfaceMapping
├── cov.nii.gz
├── cov_norm_modulate.nii.gz
├── cov_norm_modulate_ribbon.nii.gz
├── cov_ribbon.nii.gz
├── cov_ribbon_norm.nii.gz
├── cov_ribbon_norm_s5.nii.gz
├── goodvoxels.nii.gz
├── L.cov.32k_fs_LR.func.gii
├── L.cov_all.32k_fs_LR.func.gii
├── L.cov_all.native.func.gii
├── L.cov.native.func.gii
├── L.goodvoxels.32k_fs_LR.func.gii
├── L.goodvoxels.native.func.gii
├── L.mean.32k_fs_LR.func.gii
├── L.mean_all.32k_fs_LR.func.gii
├── L.mean_all.native.func.gii
├── L.mean.native.func.gii
├── mask.nii.gz
├── mean.nii.gz
├── R.cov.32k_fs_LR.func.gii
├── R.cov_all.32k_fs_LR.func.gii
├── R.cov_all.native.func.gii
├── R.cov.native.func.gii
├── R.goodvoxels.32k_fs_LR.func.gii
├── R.goodvoxels.native.func.gii
├── ribbon_only.nii.gz
├── R.mean.32k_fs_LR.func.gii
├── R.mean_all.32k_fs_LR.func.gii
├── R.mean_all.native.func.gii
├── R.mean.native.func.gii
├── SmoothNorm.nii.gz
└── std.nii.gz
-
I am a bit confused here now. Yesterday we had issues with
hcp_pre_freesurfer, you are saying those were not resolved and are now pointing to issues inhcp_icafixwhich is 5 steps down the line. Is it correct that the processing works if you run it command by command (at least up tohcp_icafix) and does not work if you use your looping script? This suggests what we have been discussing all along, there seem to be issues with your script … -
You use the
--bindparameter in intro commands but then all of a sudden it is gone … Please double check everything before running and posting here. Pretty much all issues we had to resolve to get where we are were not QuNex issues but sloppy code isseus. -
In your case, I would process this the following way:
- Execute the early prep commands over all sessions (everything up to
hcp_pre_freesurfer). - Execute
hcp_pre_freesurferin batches across all sessions (say 20 sessions at a time, depending on the capabilities of your system). - Once that is done, check the results and move on to
hcp_freesurfer.
That way you will be aware what the state of processing is and where you stand, also it is nice if you track the processing progress somewhere else (say an Excel table), so you have a clear overview of where things stand. The way you are executing things right now, there can be a huge desynchrony in processing progress and as things come crashing down you will not know what. The alternative is to run everything up to hcp_pre_freesurfer and then run every session through HCP Pipelines and keep track of that.
hcp_post_fixandhcp_dedrift_and_resampleare not needed, they are executed automatically byhcp_icafixandhcp_msmall.