Hello,
We are working with qunex container 0.97.0 to run diffusion preprocessing, and we ran into an error.
Image Exception : #63 :: No image files match: /state/partition1/home/theoweiss/qunex/qubids/sessions/006365_202304181918/hcp/006365_202304181918/Diffusion/topup/Pos_b0
We suspect that the problem is in basic_preproc_sequence.sh in line 141:
echo “Merging Pos and Neg images”
${FSLDIR}/bin/fslmerge -t ${rawdir}/Pos_b0 $(${FSLDIR}/bin/imglob ${rawdir}/Pos_b0_???.*)
My ${rawdir} contains files named “Pos_1_b0_.nii.gz" instead of "Pos_b0_.nii.gz” (no “1” in it).
We could not find any reason for this naming of these files, and would appreciate any help you can give. The logs are attached.
Thank you, Theo
[error_hcp_diffusion_006365_202304181918_2023-04-24_04.51.10.603626.log|
Log-hcp_diffusion-2023-04-24_04.51.10.583848.log (3.4 KB)
attachment](upload://ao4w08Z2FZpG56uCdgzIWX12ujJ.log) (56.5 KB)
Hi Theo,
Please switch to QuNex 0.97.3 and try again. FSL and HCP Pipelines updates that were introduced with 0.97.0 spawned a number of bugs, hopefully those were now squashed.
If you have a GPU that can run CUDA 9.1, you could also try using 0.96.2a, that version uses a bit older versions of external libraries which are much more tested and stable.
Jure
We will try the new version and let you know.
Unfortunately, we do not have GPU.
Theo
We got the same error with QuNex 0.97.3 and the log file looks the same.
However, there were new files created, and the directory tree is:
…/sessions/006400_202304241758/hcp/006400_202304241758/Diffusion/
├── data
├── eddy
│ ├── acqparams.txt
│ ├── eddy_unwarped_images.eddy_command_txt
│ ├── eddy_unwarped_images.eddy_values_of_all_input_parameters
│ ├── index.txt
│ ├── Neg.bval
│ ├── Neg.bvec
│ ├── Neg_SeriesVolNum.txt
│ ├── nodif_brain_mask.nii.gz
│ ├── Pos.bval
│ ├── Pos.bvec
│ ├── Pos_Neg.bvals
│ ├── Pos_Neg.bvecs
│ ├── Pos_Neg.nii.gz
│ ├── Pos_SeriesVolNum.txt
│ └── series_index.txt
├── rawdata
│ ├── Neg_1.bval
│ ├── Neg_1.bvec
│ ├── Neg_1.nii.gz
│ ├── Neg_SeriesCorrespVolNum.txt
│ ├── Pos_1.bval
│ ├── Pos_1.bvec
│ ├── Pos_1.nii.gz
│ └── Pos_SeriesCorrespVolNum.txt
├── reg
│ ├── nodif2T1w_initII_fast_wmedge.nii.gz
│ ├── nodif2T1w_initII_fast_wmseg.nii.gz
│ └── T1w_acpc_dc_restore_brain.nii.gz
└── topup
├── acqparams.txt
├── extractedb0.txt
├── hifib0.nii.gz
├── nodif_brain_mask.nii.gz
├── nodif_brain.nii.gz
├── Pos_Neg_b0.nii.gz
├── Pos_Neg_b0.topup_log
├── topup_Pos_Neg_b0_fieldcoef.nii.gz
├── topup_Pos_Neg_b0_field.nii.gz
└── topup_Pos_Neg_b0_movpar.txt
5 directories, 36 files
Thank you
In your previous post you said that you do not have a GPU (NVIDIA CUDA), this means that by default the command will not work as it tries to facilitate those resources. You can process without a GPU, in this case you need to use version 0.96.2a and add the --hcp_dwi_nogpu to the command call. Because of some changes in HCP/FSL the nogpu flag currently does not work with 0.97.3. A large update do diffusion processing is comming in 0.98.0 (probably released in May), this will add the nogpu flag to all DWI commands.
Jure
Hello,
Version 0.96.2a didn’t solve the problem, still the same error:
Image Exception : #63 :: No image files match: /state/partition1/home/theoweiss/qunex/qubids/sessions/006432_202305021741/hcp/006432_202305021741/Diffusion/topup/Pos_b0
The tree inside diffusion folder looks the same as well.
Thank you,
Theo
Hi,
Could you upload the full error log from the run with 0.96.2a along with the full command call.
Thanks, Jure
I ran the following command:
qunex_container run_turnkey --rawdatainput=“/state/partition1/home/theoweiss/PreProcessing/example/bids_data/” --dataformat=“BIDS” --paramfile=“/state/p1/home/theoweiss/PreProcessing/example/hcp/006432_parameters.txt” --mappingfile=“/state/partition1/home/theoweiss/PreProcessing/example/hcp/006432_mapping.txt” --workingdir=“/stition1/home/theoweiss/qunex/” --projectname=“qubids” --path=“/state/partition1/home/theoweiss/qunex/qubids” --sessions=“006432_202305021741” --sessionsfoldername=“sessions” --hcme=“userdefined” --turnkeytype=“${RUNTURNKEY_TYPE}” --container=“${QUNEX_CONTAINER}” --turnkeysteps=“${RUNTURNKEY_STEPS}” --overwritesession=“yes”
The runlog and comlog are attached.
error_hcp_diffusion_006432_202305021741_2023-05-05_02.08.11.748368.log (59.0 KB)
Log-hcp_diffusion-2023-05-05_02.08.11.717582.log (3.4 KB)
Thank you
Hi,
Could you try running the command directly, not through run_turnkey. run_turnkey usually used for chaining multiple commands together, when running only a single step it is much better to just run that. Based on your logs it should look something like:
qunex_container hcp_diffusion \
--batchfile="/state/partition1/home/theoweiss/qunex/qubids/processing/006432_parameters.txt" \
--bind="/state/partition1/home/theoweiss/qunex:/state/partition1/home/theoweiss/qunex" \
--overwrite="yes" \
--hcp_dwi_nogpu \
--container="<path to the qunex Singularity container or docker image>"
Based on what you described you have no GPU and you are not running on a high performance compute cluster through a scheduler right?
The error arises because the command is trying to use a GPU but you do not have one:
EDDY::: EddyCudaHelperFunctions::InitGpu: cudaGetDevice returned an error: cudaError_t = 38, cudaErrorName = cudaErrorNoDevice, cudaErrorString = no CUDA-capable device is detected
This is why hcp_dwi_nogpu is needed.
Jure
Hi,
I tried using your command with some changes - especially adding a scheduler:
qunex_container hcp_diffusion
–batchfile=“/state/partition1/home/theoweiss/qunex/qubids/processing/006432_parameters.txt”
–bind=“/state/partition1/home/theoweiss/qunex:/state/partition1/home/theoweiss/qunex”
–overwrite=“yes”
–hcp_dwi_nogpu
–container=“${QUNEX_CONTAINER}”
–logfolder=“${log_folder}”
–scheduler=“PBS,nodes=1:ncpus=1,walltime=48:00:00,mem=8gb,jobname=qxbids”
the directories tree at the end is different now, and I think I saw during the run a file named “pos_b0.nii.gz” in “topup” folder. But the error log file still specifies missing “pos_b0” file (with no extention). I don’t know if this “pos_b0.nii.gz” file was created in previous runs as well.
The log file is attaches, here is the directories tree:
├── data
│ ├── avg_data.idxs
│ ├── bvals
│ ├── bvals_noRot
│ ├── bvecs
│ ├── bvecs_noRot
│ ├── cnr_maps.nii.gz
│ ├── data.nii.gz
│ ├── fov_mask.nii.gz
│ ├── nodif_brain_mask.nii.gz
│ ├── nodif_brain.nii.gz
│ └── nodif.nii.gz
├── eddy
│ ├── acqparams.txt
│ ├── eddy_unwarped_images.eddy_cnr_maps.nii.gz
│ ├── eddy_unwarped_images.eddy_command_txt
│ ├── eddy_unwarped_images.eddy_movement_rms
│ ├── eddy_unwarped_images.eddy_outlier_map
│ ├── eddy_unwarped_images.eddy_outlier_n_sqr_stdev_map
│ ├── eddy_unwarped_images.eddy_outlier_n_stdev_map
│ ├── eddy_unwarped_images.eddy_outlier_report
│ ├── eddy_unwarped_images.eddy_parameters
│ ├── eddy_unwarped_images.eddy_post_eddy_shell_alignment_parameters
│ ├── eddy_unwarped_images.eddy_post_eddy_shell_PE_translation_parameters
│ ├── eddy_unwarped_images.eddy_restricted_movement_rms
│ ├── eddy_unwarped_images.eddy_rotated_bvecs
│ ├── eddy_unwarped_images.eddy_values_of_all_input_parameters
│ ├── eddy_unwarped_images.nii.gz
│ ├── index.txt
│ ├── Neg.bval
│ ├── Neg.bvec
│ ├── Neg_rotated.bvec
│ ├── Neg_SeriesVolNum.txt
│ ├── nodif_brain_mask.nii.gz
│ ├── Pos.bval
│ ├── Pos.bvec
│ ├── Pos_Neg.bvals
│ ├── Pos_Neg.bvecs
│ ├── Pos_Neg.nii.gz
│ ├── Pos_rotated.bvec
│ ├── Pos_SeriesVolNum.txt
│ └── series_index.txt
├── QC
│ ├── avg_b0_pe0.png
│ ├── avg_b0_pe1.png
│ ├── avg_b0.png
│ ├── avg_b1000.png
│ ├── avg_b2000.png
│ ├── avg_b4000.png
│ ├── cnr0000.nii.gz.png
│ ├── cnr0001.nii.gz.png
│ ├── cnr0002.nii.gz.png
│ ├── cnr0003.nii.gz.png
│ ├── qc.json
│ ├── qc.pdf
│ ├── ref_list.png
│ ├── ref.txt
│ └── vdm.png
├── rawdata
│ ├── Neg_1.bval
│ ├── Neg_1.bvec
│ ├── Neg_1.nii.gz
│ ├── Neg_SeriesCorrespVolNum.txt
│ ├── Pos_1.bval
│ ├── Pos_1.bvec
│ ├── Pos_1.nii.gz
│ └── Pos_SeriesCorrespVolNum.txt
├── reg
│ ├── nodif2T1w_initII_fast_wmedge.nii.gz
│ ├── nodif2T1w_initII_fast_wmseg.nii.gz
│ ├── nodif2T1w_initII_init.mat
│ ├── nodif2T1w_initII.mat
│ ├── nodif2T1w_initII.nii.gz
│ ├── nodif2T1w_init.nii.gz
│ ├── nodif2T1w_restore_initII.nii.gz
│ └── T1w_acpc_dc_restore_brain.nii.gz
└── topup
├── acqparams.txt
├── extractedb0.txt
├── hifib0.nii.gz
├── nodif_brain_mask.nii.gz
├── nodif_brain.nii.gz
├── Pos_Neg_b0.nii.gz
├── Pos_Neg_b0.topup_log
├── topup_Pos_Neg_b0_fieldcoef.nii.gz
├── topup_Pos_Neg_b0_field.nii.gz
└── topup_Pos_Neg_b0_movpar.txt
6 directories, 81 files
error_hcp_diffusion_006432_202305021741_2023-05-09_02.41.59.572828.log (58.3 KB)
Thank you
Theo,
I believe that pos_b0.nii.gz is a temporary file that gets removed at the end. I checked a couple of my studies and it was also missing from the final results (where hcp_diffusion was successfully executed). I looked at one of the successful logs from the same version as yours (0.96.2a, attached below) and everything looks “the same” until your error, the command call also looks fine, so this might be a HCP Pipelines issue or an issue with your data. Maybe you can try looking through or asking at https://groups.google.com/a/humanconnectome.org/g/hcp-users and see if someone saw this before.
We are working on a big diffusion rework that also updates all of the used tools in the background, expected release is this month. If this is not super pressing, you can maybe wait for this update.
Also, when pasting commands and long blocks of code or outputs (e.g., the folder structure), please encapsulate them with triple back tick (`) on the forum so it is easier to read, e.g.:
├── data
│ ├── avg_data.idxs
│ ├── bvals
│ ├── bvals_noRot
│ ├── bvecs
│ ├── bvecs_noRot
│ ├── cnr_maps.nii.gz
│ ├── data.nii.gz
│ ├── fov_mask.nii.gz
│ ├── nodif_brain_mask.nii.gz
│ ├── nodif_brain.nii.gz
│ └── nodif.nii.gz
vs
├── data
│ ├── avg_data.idxs
│ ├── bvals
│ ├── bvals_noRot
│ ├── bvecs
│ ├── bvecs_noRot
│ ├── cnr_maps.nii.gz
│ ├── data.nii.gz
│ ├── fov_mask.nii.gz
│ ├── nodif_brain_mask.nii.gz
│ ├── nodif_brain.nii.gz
│ └── nodif.nii.gz
Cheers, Jure
done_hcp_diffusion_S4453_P49_JV_2022-12-16_05.19.22.853993.log (119.3 KB)
I’ll wait for the next update.
Sorry about the code pasting, and thanks a lot for your help,
Theo
Hi, we will be releasing a new version that should fix this soon. Please test it and let us know if there are any issues.
Jure
Thank you for the message,
I’ll try it,
Theo
Sorry about the delay,
The same issues occurred with the new version. we suspect it has something to do with our FSL version, which we cannot update for now because it is in use in running studies.
The pipeline is still very helpful and we will use the other parts of it for now.
Thanks a lot for the support,
Theo