Hi,
We have recently started running the diffusion preprocessing pipeline (without GPU).
It ran successfully with the following command, for subject 10001:
\qunex_container hcp_diffusion \
\ "sessions="${SESSIONS}-- \
\ "mappingfile="${INPUT_MAPPING_FILE}-- \
\ "container="${QUNEX_CONTAINER}-- \
\"dockeropt="-v ${BIND_FOLDER}:${BIND_FOLDER}-- \
\ "sessionsfolder="${STUDY_FOLDER}/sessions-- \
\ "batchfile="${STUDY_FOLDER}/processing/batch.txt-- \
\ parsessions=3-- \
\ "hcp_dwi_negdata="10001_DWI_dir6_PA.nii.gz-- \
\ "hcp_dwi_posdata="10001_DWI_dir185_AP.nii.gz-- \
\ hcp_dwi_combinedata=2-- \
\ hcp_dwi_selectbestb-- \
\ hcp_nogpu-- \
\ extra-eddy-arg=--niter=8-- \
extra-eddy-arg=--fwhm=10,8,6,4,2,0,0,0 0--
We wish to run this command on several participants at the same time. How should we modify the command so instead of the hard-coded subject number, we can use a variable that changes to the relevant subject number on every run?
Many thanks!
Yael
Hi Yael,
if your batch file is correctly prepared and it includes parameters on top, followed by the data about all sessions and their images, then you can simply skip the --sessions="${SESSIONS}" parameter and QuNex will run the processing over all sessions inside the batch file. As you have parsessions set to 3, it will process 3 sessions in parallel. Once one of the three finishes, it will automatically pick the next one. Also, the hardcoded negdata and posdata can be skipped in this case, as QuNex will automatically pull it for each session from the batch file.
Let me know if this works. You can also upload your batch file here so I can check it looks good.
Best, Jure
Hi Jure! I’m working with Yael on the matter.
When we tried running without the negdata and posdata it fails immediately. Looking at the logs of one participant, these lines appear:
–posData=“EMPTY@/home/docker/volumes/hcppipelines/NogaArmyStudy/soldiers/sessions/SNBB_YBH_BrainBank_20240411_1114/hcp/SNBB_YBH_BrainBank_20240411_1114/unprocessed/Diffusion/SNBB_YBH_BrainBank_20240411_1114_DWI_dir6_PA.nii.gz” \
–negData=“/home/docker/volumes/hcppipelines/NogaArmyStudy/soldiers/sessions/SNBB_YBH_BrainBank_20240411_1114/hcp/SNBB_YBH_BrainBank_20240411_1114/unprocessed/Diffusion/SNBB_YBH_BrainBank_20240411_1114_DWI_dir185_AP.nii.gz@EMPTY” \
Mon Jul 27 10:37:18 UTC 2026:DiffPreprocPipeline_PreEddy.sh: ERROR: Wrong Input! No pairs of phase encoding directions have been found!
Mon Jul 27 10:37:18 UTC 2026:DiffPreprocPipeline_PreEddy.sh: ERROR: Wrong Input! No pairs of phase encoding directions have been found!
Mon Jul 27 10:37:18 UTC 2026:DiffPreprocPipeline_PreEddy.sh: ERROR: At least one pair is needed!
Mon Jul 27 10:37:18 UTC 2026:DiffPreprocPipeline_PreEddy.sh: ERROR: At least one pair is needed!
meaning, it doesn’t correctly associate XXX_DWI_dir6_PA with negdata and XXX_DWI_dir185_AP with posdata, or at least that’s what we think goes wrong.
hence the need to use a joker/please holder as Yael mentioned. or should the negdata and posdata be specified in the batch file? if so, how?
Hi!
You do not have a full AP/PA pair with the same number of dirs, which is the HCP’s default. I believe you need to set --hcp_dwi_combinedata=2 in such cases.
It has been a while since I did this, so there is a slight chance that I am wrong. Maybe best to check with the HCP team on their forum => https://groups.google.com/a/humanconnectome.org/g/hcp-users. They should be able to give you optimal parameters for your data.
Best, Jure
We did indeed use this flag. here’s how we ran the group:
time qunex_container hcp_diffusion
–sessionsfolder=“${STUDY_FOLDER}/sessions”
–batchfile=“${STUDY_FOLDER}/processing/batch.txt”
–sessions=“${SESSIONS}”
–container=“${QUNEX_CONTAINER}”
–dockeropt=“-v ${BIND_FOLDER}:${BIND_FOLDER}”
–parsessions=3
–hcp_dwi_combinedata=2
–hcp_dwi_selectbestb
–hcp_nogpu
–extra-eddy-arg=“–niter=8”
–extra-eddy-arg=“–fwhm=10,8,6,4,2,0,0,0”
We will try the HCP forum, thanks!
Hi,
You have a typo in hcp_dwi_selectbestb it should be hcp_dwi_selectbestb0. I believe this is the issue as your data needs this setting.
Sorry for missing this and also the hcp_dwi_combinedata parameter, I now see it is indeed there. I probably missed it because your code parts are a bit hard to read. Please encompass code into triple back ticks (`) so it is easier to read, and makes our work easier. For example:
time qunex_container hcp_diffusion
–sessionsfolder=“${STUDY_FOLDER}/sessions”
–batchfile=“${STUDY_FOLDER}/processing/batch.txt”
–sessions=“${SESSIONS}”
–container=“${QUNEX_CONTAINER}”
–dockeropt=“-v ${BIND_FOLDER}:${BIND_FOLDER}”
–parsessions=3
–hcp_dwi_combinedata=2
–hcp_dwi_selectbestb
–hcp_nogpu
–extra-eddy-arg=“–niter=8”
–extra-eddy-arg=“–fwhm=10,8,6,4,2,0,0,0”
vs
time qunex_container hcp_diffusion
–sessionsfolder=“${STUDY_FOLDER}/sessions”
–batchfile=“${STUDY_FOLDER}/processing/batch.txt”
–sessions=“${SESSIONS}”
–container=“${QUNEX_CONTAINER}”
–dockeropt=“-v ${BIND_FOLDER}:${BIND_FOLDER}”
–parsessions=3
–hcp_dwi_combinedata=2
–hcp_dwi_selectbestb
–hcp_nogpu
–extra-eddy-arg=“–niter=8”
–extra-eddy-arg=“–fwhm=10,8,6,4,2,0,0,0”
Best, Jure