[RESOLVED] Issues with dwi_probtrackx_dense_gpu

Dear Jure:

I have been trying to run dwi_probtrackx_dense_gpu using the 0.99.2d release but I got the following error:

# Generated by QuNex 0.99.2 on 2024-02-09_09.49.18.115382

-- qunex.sh: Specified Command-Line Options - Start --
   Study Folder: /home/ehui/qunex/cimt
   Sessions Folder: /home/ehui/qunex/cimt/sessions
   Session: 024A
   probtrackX GPU scripts Folder: /opt/qunex/bash/qx_utilities/diffusion_tractography_dense/tractography_gpu_scripts
   Compute Matrix1: no
   Compute Matrix3: yes
   Number of samples for Matrix1: 10000
   Number of samples for Matrix3: 3000
   Distance correction: no
   Store streamlines length: no
   Overwrite prior run: yes
   No GPU: no
-- qunex.sh: Specified Command-Line Options - End --

^[[32m ------------------------- Start of work -------------------------------- ^[[0m

^[[32m    --- probtrackX GPU for session 024A... ^[[0m

^[[31m  --- Removing existing Probtrackxgpu Matrix3 dense run for 024A... ^[[0m

^[[32m Checking if ProbtrackX Matrix 3 and dense connectome was completed on 024A... ^[[0m

^[[32m ProbtrackX Matrix 3 solution and dense connectome incomplete for 024A. Starting run with 3000 samples... ^[[0m

Running the following probtrackX GPU command:


   /opt/qunex/bash/qx_utilities/diffusion_tractography_dense/tractography_gpu_scripts/run_matrix3.sh /home/ehui/qunex/cimt/sessions 024A 3000 no no no


-- Queueing Probtrackx

/home/ehui/qunex/cimt/sessions/024A/hcp/024A/MNINonLinear/Results/Tractography/commands_Mat3.sh: line 1: /opt/fsl/fsl/bin/probtrackx2_gpu10.1: No such file or directory

-- Queueing Post-Matrix 3 Calls

Kindly note that below are the CUDA that I have installed on my workstation:
CUDA-10.1, CUDA-10.2, CUDA-11.8

Many thanks for your help as always!


Hi Ed,

Could you please provide the full command call that you used. I believe this is a minor thing, we just need to set the correct version of CUDA that will get used.

Best, Jure

Hi Jure:

Absolutely, please see below:


#cd ~/qunex

#sudo docker login gitlab.qunex.yale.edu:5002 -u ehui
#sudo docker pull gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:0.93.6

export SDIR="${HOME}"
export STUDY_NAME="cimt"
export WORK_DIR="${SDIR}/qunex"
export QUNEX_CONTAINER="${HOME}/qunex/qunexcontainer/qunex_suite-0.99.2d.sif" 
export RAW_DATA="${WORK_DIR}/data"
export INPUT_BATCH_FILE="${RAW_DATA}/006_batch.txt"
export INPUT_MAPPING_FILE="${RAW_DATA}/006_mapping__.txt" #pci.txt"
export SESSIONS="024A" 

qunex_container dwi_probtrackx_dense_gpu \
  --sessionsfolder="${WORK_DIR}/${STUDY_NAME}/sessions" \
  --sessions="${SESSIONS}" \
  --omatrix3="yes" \
  --overwrite="yes" \
  --container="${QUNEX_CONTAINER}" \
  --bash_pre="module load CUDA/10.1" \
  --bash_post="export DEFAULT_CUDA_VERSION=10.1" \
  --bind="/usr/local/cuda-10.1/:/usr/local/cuda/" \

Thanks a lot,

Please try replacing your command with:

qunex_container dwi_probtrackx_dense_gpu \
  --sessionsfolder="${WORK_DIR}/${STUDY_NAME}/sessions" \
  --sessions="${SESSIONS}" \
  --omatrix3="yes" \
  --overwrite="yes" \
  --container="${QUNEX_CONTAINER}" \
  --bash_pre="module load CUDA/10.2" \
  --bind="/usr/local/cuda-10.2/:/usr/local/cuda/" \

10.2 is the default used CUDA version and the one we tested the most. I think this should work. Let me know how it goes.

Hello Jure:

My apologies for the late reply.

It works! Many thanks for your help indeed!