[RESOLVED] Eddy failed with message EDDY::eddy.cpp

[yzhong]

Hi all,

Description:

When processing DWI data (dwi_legacy_gpu step), some data encountered an EDDY error, but hcp_prefreesurfer, hcp_freesurfer, and hcp_postfreesurfer all ran successfully. What could be causing this error?

Call:

		qunex_container hcp_pre_freesurfer \
		    --sessionsfolder="${WORK_DIR}/${STUDY_NAME}/sessions" \
		    --batchfile="${WORK_DIR}/${STUDY_NAME}/processing/batch2.txt" \
		    --parsessions="${PARSES}" \
		    --hcp_processing_mode="${hcp_processing_mode}" \
            --hcp_avgrdcmethod="NONE" \
		    --hcp_t2="${hcp_t2}" \
		    --hcp_topupconfig="b02b0.cnf" \
            --hcp_prefs_template_res="1" \
		    --hcp_t1samplespacing="0.0000071" \
		    --hcp_t2samplespacing="0.0000021" \
		    --hcp_seechospacing="0.000580009" \
		    --hcp_unwarpdir="z" \
		    --hcp_seunwarpdir="y" \
		    --hcp_sephaseneg="AP" \
		    --hcp_sephasepos="PA" \
		    --hcp_seunwarpdir="y" \
            --hcp_bold_dcmethod="NONE" \
		    --bind="${WORK_DIR}:${WORK_DIR}" \
            --container="${QUNEX_CONTAINER}" \
            --overwrite="yes"
		;;

        qunex_container hcp_freesurfer \
                    --sessionsfolder="${WORK_DIR}/${STUDY_NAME}/sessions" \
                    --batchfile="${WORK_DIR}/${STUDY_NAME}/processing/batch2.txt" \
                    --parsessions="${PARSES}" \
                    --hcp_processing_mode="${hcp_processing_mode}" \
                    --hcp_t2="${hcp_t2}" \
		            --hcp_freesurfer_home="${FREESURFER_HOME}" \
                    --bind="${WORK_DIR}:${WORK_DIR}" \
                    --container="${QUNEX_CONTAINER}" \
                    --overwrite="yes"
                ;;
	

         qunex_container hcp_post_freesurfer \
                    --sessionsfolder="${WORK_DIR}/${STUDY_NAME}/sessions" \
                    --batchfile="${WORK_DIR}/${STUDY_NAME}/processing/batch2.txt" \
                    --parsessions="${PARSES}" \
                    --hcp_processing_mode="${hcp_processing_mode}" \
                    --hcp_t2="${hcp_t2}" \
		            --hcp_regname="MSMSulc" \
	 	            --hcp_grayordinatesres="2" \
		            --hcp_hiresmesh="164" \
		            --hcp_lowresmesh="32" \
                    --bind="${WORK_DIR}:${WORK_DIR}" \
                    --container="${QUNEX_CONTAINER}" \
                    --overwrite="yes"

		  qunex_container dwi_legacy_gpu \
			    --sessionsfolder="${WORK_DIR}/${STUDY_NAME}/sessions" \
			    --batchfile="${WORK_DIR}/${STUDY_NAME}/processing/batch2.txt" \
			    --echospacing="0.69" \
			    --PEdir="2" \
			    --TE="2.46"\
			    --unwarpdir="y-" \
			    --diffdatasuffix="DWI_dir64_AP" \
			    --usefieldmap="no" \
			    --scanner="siemens" \
			    --overwrite="yes" \
			    --bind="${WORK_DIR}:${WORK_DIR}" \
			    --container="${QUNEX_CONTAINER}" \
			    --nv;

Logs:

Here are my log files:

error_dwi_legacy_gpu_OAS30001_d0757_2025-11-13_15.11.15.765240.log (16.2 KB)

done_hcp_pre_freesurfer_OAS30001_d0757_2025-11-13_11.42.16.060570.log (29.5 KB)

done_hcp_post_freesurfer_OAS30001_d0757_2025-11-13_12.31.34.211617.log (1.8 MB)

Thanks for the kindly help!

Best,

Acacius

[demsarjure]

Hi Acacius,

Seems like some kind of a data issue, this seems to be the more explicit error.

EDDY::ReplacementManager::Update: Mismatched DiffStatsVector object

This happens in the Checking for outliers eddy step.

I would try the following:

1. Use QuNex 1.4.0.
2. Try to run without a gpu.
3. Also try on another session from this study to check if this is a one off or if all sessions are probelmatic.

Best, Jure

[yzhong]

Thanks. I will try to do so.

Best,

Acacius

[yzhong]

Hi Jure,

I have tried both using Qunex 1.4.0 and with/without GPU. The errors are the same. In addition, only few sessions can be run successfully but most of them are failed in this step. I have checked the quality of the image, all of them are right. What could be the cause of this error?

Here also provides an failed session for ref:

OAS30001_d0757.zip

Best,

Acacius

[demsarjure]

Hi,

What is weird is that some sessions work.

Does anything stick out at the outgoing eddy call:

/opt/fsl/fsl/bin/eddy_cuda10.2 --imain=/mnt/ssd2/yumingz/OASIS3//try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d0757_DWI_dir64_AP --mask=/mnt/ssd2/yumingz/OASIS3//try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/rawdata/OAS30001_d0757_DWI_dir64_AP_nodif_brain_mask --acqp=/mnt/ssd2/yumingz/OASIS3//try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/acqparams/OAS30001_d0757_DWI_dir64_AP/acqparams.txt --index=/mnt/ssd2/yumingz/OASIS3//try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/acqparams/OAS30001_d0757_DWI_dir64_AP/index.txt --bvecs=/mnt/ssd2/yumingz/OASIS3//try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d0757_DWI_dir64_AP.bvec --bvals=/mnt/ssd2/yumingz/OASIS3//try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d0757_DWI_dir64_AP.bval --fwhm=10,0,0,0,0 --ff=10 --nvoxhp=2000 --flm=quadratic --out=/mnt/ssd2/yumingz/OASIS3//try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/eddy/OAS30001_d0757_DWI_dir64_AP_eddy_corrected --data_is_shelled --repol -v

Can you upload the batch or session_hcp file as well please so I can see what the data looks like. We made some upgrades to the HCP Diffusion pipeline to support additional DWI acquisitions, maybe now you could use that.

One thing you could try is to run eddy without the --data_is_shelled flag. But there is currently no way to do this with legacy command, only with HCP one. If you want to try this out, I can prepare a script and instructions on how to do it.

Since it crashes in outlier detection another 2 relevant settings might be:

• --ff=10, which sets the hyperparameter for the outlier detection model (ff = fudge factor),
• --repol, which replaces outlier slices rather than simply flagging them.

In the next version, I will code a parameter that will allow you to set and tweak these eddy parameters yourself. The default will be as is, but you will be able to change them.

Best, Jure

[yzhong]

Hi Jure,

Thanks for your explanation. I have tried to use HCP command to run, but it still failed. And here is my batch file for your reference. In this file, only OAS30003_d2682,OAS30003_d4954,OAS30001_d4467 session can be processed sucessfully.

batch2.txt (12.8 KB)

Many thanks!

Best, Acacius

[demsarjure]

Yes, hcp_diffusion needs a fieldmap, so this won’t work.

I am out of ideas. Let us wait for QuNex 1.4.1 which will allow us to tweak eddy parameters and maybe that will help. Up to eddy, everything looks good.

Best, Jure

[yzhong]

OK, always thanks for your kindly help!

Best, Acacius

[yzhong]

Hi Jure,

I attempted to run eddy directly within Qunex Singularity, but encountered an error resembling garbled characters. Could this error stem from the data itself rather than the parameters?

Apptainer> $FSLDIR/bin/eddy \

–imain=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d0757_DWI_dir64_AP.nii.gz
–mask=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/rawdata/OAS30001_d0757_DWI_dir64_AP_nodif_brain_mask
–acqp=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/acqparams/OAS30001_d0757_DWI_dir64_AP/acqparams.txt
–index=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/acqparams/OAS30001_d0757_DWI_dir64_AP/index.txt
–bvecs=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d0757_DWI_dir64_AP.bvec
–bvals=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d0757_DWI_dir64_AP.bval
–out=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/eddy/OAS30001_d0757_DWI_dir64_AP_eddy_corrected
–data_is_shelled
-v
Reading images
Performing volume-to-volume registration
Running Register
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 0, Total mss = 214.132
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 1, Total mss = 155.472
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 2, Total mss = 152.935
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 3, Total mss = 152.728
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 4, Total mss = 152.472
Running sm.ApplyB0LocationReference
Running Register
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 21.827676
3.828026
-0.968406
9.993975
-0.290176
23.161311
-45.308476
21.593793
0.139647
-19.083001
22.313109
45.019599

Calculating parameter updates
Iter: 0, Total mss = 746.245535
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 7.114375
3.639971
-0.965886
-14.788797
0.850591
8.382871
6.766662
7.270483
15.473105
5.928752
12.914583
21.348877

Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 8.526673
17.305372
-0.392678
-34.081441
-29.721117
6.716848
4.668277
1.004080
7.513318
-12.289331
50.538131
5.705648

Calculating parameter updates
Iter: 1, Total mss = 817.819453
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 6.544746
3.407133
-1.002646
4.001862
1.039864
7.787582
-10.052452
6.646038
6.822640
-1.298378
8.159269
25.462813

Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 4.488513
2.623871
-1.061775
40.825844
-2.746121
5.681130
-18.545067
4.941389
11.093012
-2.962520
18.357497
17.146166

Calculating parameter updates
Iter: 2, Total mss = 728.695783
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 11.408223
4.366914
-0.960240
0.655497
0.719262
12.829215
-13.250172
12.003692
13.000057
-12.349855
24.136301
36.111216

Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 5.895589
3.869134
-1.031149
64.455534
-5.771210
7.614920
-53.385761
6.544221
7.747121
-10.332651
27.978473
14.828423

Calculating parameter updates
Iter: 3, Total mss = 777.852793
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 9.253500
4.925911
-0.932221
-15.647024
3.654969
10.613693
-17.041611
9.516896
10.633439
6.711903
10.518202
33.500305

Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 20.907997
7.895965
-0.402846
4.931095
-5.936272
18.866633
-8.529236
16.883703
3.540919
-5.199254
13.986541
17.830126

Calculating parameter updates
Iter: 4, Total mss = 856.595136
Running sm.ApplyDWILocationReference

▒▒▒G|

EDDY::: Eddy failed with message ▒IMX

Best, Acacius

[demsarjure]

Hi,

These garbled characters usually pop up when there are CUDA/GPU issues. How did you enter the container. Please paste the full command and I will let you know if something sticks out.

Best, Jure

[yzhong]

Hi Jure

Here is my full command:

singularity shell -B /mnt/ssd2/yumingz/OASIS3:/mnt/ssd2/yumingz/OASIS3 qunex_suite-0.99.2d.sif

Apptainer> /opt/fsl/fsl-6.0.7.1/bin/eddy --imain=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d7_DWI_dir64_AP.nii.gz --mask=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/rawdata/OAS30001_d0757_DWI_dirAP_nodif_brain_mask --acqp=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/acqparams/OAS30001_d0757_DWI_dirAP/acqparams.txt --index=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/acqparams/OAS30001_d0757_DWI_dir64/index.txt --bvecs=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d0757_DWI_dir64_AP.bvec --bvalsnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d0757_DWI_dir64_AP.bval --fwhm=10,0,0,0,0 --ff=10 --xhp=2000 --flm=quadratic --out=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/eddy/OAS30001_d0757_DWI_dir6P_eddy_corrected --data_is_shelled --repol -v
/usr/bin/env: fslpython: Permission denied
Apptainer> mkdir -p /tmp/fsl_bin
Apptainer> ln -sf $(which python3) /tmp/fsl_bin/fslpython
Apptainer> export PATH=“/tmp/fsl_bin:$PATH”
Apptainer> /opt/fsl/fsl-6.0.7.1/bin/eddy \

–imain=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d0757_DWI_dir64_AP.nii.gz
–mask=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/rawdata/OAS30001_d0757_DWI_dir64_AP_nodif_brain_ma
–acqp=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/acqparams/OAS30001_d0757_DWI_dir64_AP/acqparams.tx
–index=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/acqparams/OAS30001_d0757_DWI_dir64_AP/index.txt
–bvecs=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d0757_DWI_dir64_AP.bvec
–bvals=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d0757_DWI_dir64_AP.bval
–out=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/eddy/OAS30001_d0757_DWI_dir64_AP_eddy_corrected
–data_is_shelled
-v
Traceback (most recent call last):
File “/opt/fsl/fsl-6.0.7.1/bin/eddy”, line 98, in
main()
File “/opt/fsl/fsl-6.0.7.1/bin/eddy”, line 78, in main
fsldir = os.environ[‘FSLDIR’]
File “/usr/lib64/python3.6/os.py”, line 669, in getitem
raise KeyError(key) from None
KeyError: ‘FSLDIR’

Apptainer> export FSLDIR=/opt/fsl/fsl-6.0.7.1
Apptainer> export PATH=$FSLDIR/bin:$PATH
Apptainer> mkdir -p /tmp/fsl_bin
Apptainer> ln -sf $(which python3) /tmp/fsl_bin/fslpython
Apptainer> export PATH=“/tmp/fsl_bin:$PATH”
Apptainer> $FSLDIR/bin/eddy
–imain=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d0757_DWI_dir64_AP.nii.gz
–mask=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/rawdata/OAS30001_d0757_DWI_dir64_AP_nodif_brain_ma
–acqp=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/acqparams/OAS30001_d0757_DWI_dir64_AP/acqparams.tx
–index=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/acqparams/OAS30001_d0757_DWI_dir64_AP/index.txt
–bvecs=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d0757_DWI_dir64_AP.bvec
–bvals=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d0757_DWI_dir64_AP.bval
–out=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/eddy/OAS30001_d0757_DWI_dir64_AP_eddy_corrected
–data_is_shelled
-v
Reading images
Performing volume-to-volume registration
Running Register
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 0, Total mss = 214.132
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 1, Total mss = 155.472
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 2, Total mss = 152.935
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 3, Total mss = 152.728
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 4, Total mss = 152.472
Running sm.ApplyB0LocationReference
Running Register
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 21.827676
3.828026
-0.968406
9.993975
-0.290176
23.161311
-45.308476
21.593793
0.139647
-19.083001
22.313109
45.019599

Calculating parameter updates
Iter: 0, Total mss = 746.245535
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 7.114375
3.639971
-0.965886
-14.788797
0.850591
8.382871
6.766662
7.270483
15.473105
5.928752
12.914583
21.348877

Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 8.526673
17.305372
-0.392678
-34.081441
-29.721117
6.716848
4.668277
1.004080
7.513318
-12.289331
50.538131
5.705648

Calculating parameter updates
Iter: 1, Total mss = 817.819453
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 6.544746
3.407133
-1.002646
4.001862
1.039864
7.787582
-10.052452
6.646038
6.822640
-1.298378
8.159269
25.462813

Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 4.488513
2.623871
-1.061775
40.825844
-2.746121
5.681130
-18.545067
4.941389
11.093012
-2.962520
18.357497
17.146166

Calculating parameter updates
Iter: 2, Total mss = 728.695783
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 11.408223
4.366914
-0.960240
0.655497
0.719262
12.829215
-13.250172
12.003692
13.000057
-12.349855
24.136301
36.111216

Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 5.895589
3.869134
-1.031149
64.455534
-5.771210
7.614920
-53.385761
6.544221
7.747121
-10.332651
27.978473
14.828423

Calculating parameter updates
Iter: 3, Total mss = 777.852793
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 9.253500
4.925911
-0.932221
-15.647024
3.654969
10.613693
-17.041611
9.516896
10.633439
6.711903
10.518202
33.500305

Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 20.907997
7.895965
-0.402846
4.931095
-5.936272
18.866633
-8.529236
16.883703
3.540919
-5.199254
13.986541
17.830126

Calculating parameter updates
Iter: 4, Total mss = 856.595136
Running sm.ApplyDWILocationReference

▒▒▒G|

EDDY::: Eddy failed with message ▒IMX
Apptainer> /opt/fsl/fsl-6.0.7.1/bin/eddy
–mask=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/rawdata/OAS30001_d0757_DWI_dir64_AP_nodif_brain_mask

–imain=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d0757_DWI_dir64_AP.nii.gz
–mask=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/rawdata/OAS30001_d0757_DWI_dir64_AP_nodif_brain_ma
–acqp=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/acqparams/OAS30001_d0757_DWI_dir64_AP/acqparams.tx
–index=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/acqparams/OAS30001_d0757_DWI_dir64_AP/index.txt
–bvecs=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d0757_DWI_dir64_AP.bvec
–bvals=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/OAS30001_d0757_DWI_dir64_AP.bval
–out=/mnt/ssd2/yumingz/OASIS3/try/sessions/OAS30001_d0757/hcp/OAS30001_d0757/Diffusion/eddy/OAS30001_d0757_DWI_dir64_AP_eddy_corrected
–data_is_shelled

▒▒▒▒{

EDDY::: Eddy failed with message ▒_▒▒X

[demsarjure]

@yzhong

singularity shell -B /mnt/ssd2/yumingz/OASIS3:/mnt/ssd2/yumingz/OASIS3 qunex_suite-0.99.2d.sif

You need to add the --nv flag to link host CUDA libraries with the container. However, I am not sure whether that is needed since you are using eddy and not eddy_cuda. I also add the --cleanenv flag to prevent environment leakeage from host to container.

There is an error saying /usr/bin/env: fslpython: Permission denied. Try the following:

singularity run --bind /mnt/ssd2/yumingz/OASIS3:/mnt/ssd2/yumingz/OASIS3 --nv --cleanenv qunex_suite-0.99.2d.sif bash

You can also try omittin bash at the end. This shold enter the container and setup the QuNex enviroment in there.

If this does not work you can try

singularity shell --bind /mnt/ssd2/yumingz/OASIS3:/mnt/ssd2/yumingz/OASIS3 --nv --cleanenv qunex_suite-0.99.2d.sif

# once inside the container run
source /opt/qunex/env/qunex_environment.sh

Note that -B and --bind are the same.

Best, Jure

[yzhong]

Hi, Jure

I have tried these command to call the eddy function:

1. singularity run --bind /mnt/ssd2/yumingz/OASIS3:/mnt/ssd2/yumingz/OASIS3 --nv --cleanenv qunex_suite-0.99.2d.sif bash
2. singularity run --bind /mnt/ssd2/yumingz/OASIS3:/mnt/ssd2/yumingz/OASIS3 --nv --cleanenv qunex_suite-0.99.2d.sif
3. singularity shell --bind /mnt/ssd2/yumingz/OASIS3:/mnt/ssd2/yumingz/OASIS3 --nv --cleanenv qunex_suite-0.99.2d.sif
   source /opt/qunex/env/qunex_environment.sh

But unfortunately, all of them are failed at the same step:

Reading images
Performing volume-to-volume registration
Running Register

…Allocated GPU # 0…
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 0, Total mss = 223.703
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 1, Total mss = 155.643
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 2, Total mss = 152.944
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 3, Total mss = 152.739
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 4, Total mss = 152.491
Running sm.ApplyB0LocationReference
Running Register
Loading prediction maker
Evaluating prediction maker model
Checking for outliers
Calculating parameter updates
Iter: 0, Total mss = 702.504
Loading prediction maker
Evaluating prediction maker model
Checking for outliers
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 1, Total mss = 691.244
Loading prediction maker
Evaluating prediction maker model
Checking for outliers
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 2, Total mss = 740.555
Loading prediction maker
Evaluating prediction maker model
Checking for outliers
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 3, Total mss = 694.837
Loading prediction maker
Evaluating prediction maker model
Checking for outliers
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 4, Total mss = 881.004
Running sm.ApplyDWILocationReference

▒zD

▒zDY::: Eddy failed with message ▒gD

Reading images
Performing volume-to-volume registration
Running Register

…Allocated GPU # 0…
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 0, Total mss = 223.703
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 1, Total mss = 155.643
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 2, Total mss = 152.944
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 3, Total mss = 152.739
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 4, Total mss = 152.491
Running sm.ApplyB0LocationReference
Running Register
Loading prediction maker
Evaluating prediction maker model
Checking for outliers
Calculating parameter updates
Iter: 0, Total mss = 720.946
Loading prediction maker
Evaluating prediction maker model
Checking for outliers
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 1, Total mss = 676.286
Loading prediction maker
Evaluating prediction maker model
Checking for outliers
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 2, Total mss = 842.536
Loading prediction maker
Evaluating prediction maker model
Checking for outliers
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 3, Total mss = 883.626
Loading prediction maker
Evaluating prediction maker model
Checking for outliers
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 4, Total mss = 844.536
Running sm.ApplyDWILocationReference

▒q▒p

EDDY::: Eddy failed with message ▒q▒p

Reading images
Performing volume-to-volume registration
Running Register

…Allocated GPU # 0…
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 0, Total mss = 223.703
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 1, Total mss = 155.643
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 2, Total mss = 152.944
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 3, Total mss = 152.739
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 4, Total mss = 152.491
Running sm.ApplyB0LocationReference
Running Register
Loading prediction maker
Evaluating prediction maker model
Checking for outliers
Calculating parameter updates
Iter: 0, Total mss = 792.591
Loading prediction maker
Evaluating prediction maker model
Checking for outliers
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 1, Total mss = 862.77
Loading prediction maker
Evaluating prediction maker model
Checking for outliers
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 2, Total mss = 850.976
Loading prediction maker
Evaluating prediction maker model
Checking for outliers
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 3, Total mss = 824.243
Loading prediction maker
Evaluating prediction maker model
Checking for outliers
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 4, Total mss = 867.43
Running sm.ApplyDWILocationReference

▒▒t6v

EDDY::: Eddy failed with message ▒▒t6v

And I also put this error in the fsl forum, they gave me the reason below:

There used to be a bug in the error reporting, which is the reason that you get the incomprehensible error message. Can I please ask you to re-run the latest version of eddy so that you get a proper error message, which will help us pinpoint the problem. Once we know the problem you can hopefully go back to your old eddy version if you would prefer to use the same version for all of your data.

I would like to consult you on whether there is a way to install the latest version of FSL in a container for testing without disrupting the existing environment.

Best

Acacius

[demsarjure]

Well, you are using a super old QuNex container (0.99.2d). The latest one (1.4.0) has the latest or close to the latest FSL.

Best, Jure

[yzhong]

Thanks. I used v1.4.0 container to call eddy, now the errors are clear and shown below:

...................Allocated GPU # 0...................
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 0, Total mss = 223.703
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 1, Total mss = 155.643
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 2, Total mss = 152.946
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 3, Total mss = 152.734
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 4, Total mss = 152.485
Running sm.ApplyB0LocationReference
Running Register
Loading prediction maker
Evaluating prediction maker model
Checking for outliers
EDDY::ReplacementManager::Update: Mismatched DiffStatsVector object
EDDY::EddyHelperClasses.cpp:::  void EDDY::ReplacementManager::Update(const EDDY::DiffStatsVector&):  Exception thrown
EDDY::cuda/EddyInternalGpuUtils.cu:::  static void EDDY::EddyInternalGpuUtils::detect_outliers(const EDDY::EddyCommandLineOptions&, EDDY::ScanType, std::shared_ptr<EDDY::DWIPredictionMaker>, const NEWIMAGE::volume<float>&, const EDDY::ECScanManager&, unsigned int, unsigned int, EDDY::ReplacementManager&, EDDY::DiffStatsVector&):  Exception thrown
EDDY::cuda/EddyGpuUtils.cu:::  static EDDY::DiffStatsVector EDDY::EddyGpuUtils::DetectOutliers(const EDDY::EddyCommandLineOptions&, EDDY::ScanType, std::shared_ptr<EDDY::DWIPredictionMaker>, const NEWIMAGE::volume<float>&, const EDDY::ECScanManager&, EDDY::ReplacementManager&):  Exception thrown
EDDY::eddy.cpp:::  EDDY::ReplacementManager* EDDY::Register(const EDDY::EddyCommandLineOptions&, EDDY::ScanType, unsigned int, const std::vector<float, std::allocator<float> >&, EDDY::SecondLevelECModelType, bool, EDDY::ECScanManager&, EDDY::ReplacementManager*, NEWMAT::Matrix&, NEWMAT::Matrix&):  Exception thrown
EDDY::: Eddy failed with message EDDY::eddy.cpp:::  EDDY::ReplacementManager* EDDY::DoVolumeToVolumeRegistration(const EDDY::EddyCommandLineOptions&, EDDY::ECScanManager&):  Exception thrown

Do you have any idea about this error? Thanks again!

Best, Acacius

[demsarjure]

It seems like some issues in the outlier detection part of eddy. Now that you are manually triggering eddy, you could try removing the --data_is_shelled parameter, this one ofen caused issues with some users. Also there are two parameters that work with outliers --ff=10 and --repol. You can try changing or removing them. Unfortunately, I do not completely know how these work, the FSL mailing list is probably a better location for such an in-depth discussion.

In QuNex 1.4.1 (to be relesaed over the next 2 weeks or so), you will be able to tweak these parameters through the command call. But right now that functionality is not supported.

Best, Jure

[yzhong]

OK, many thanks for your kindly help!

Best

Acacius

[demsarjure]

Hi Acacius,

Any updated from FSL folks regarding this?

Best, Jure

[yzhong]

Hi Jure,

They asked the files of the error subjects yesterday. Hope to receive an update within the next couple of days.

Best, Acacius