[RESOLVED] Probtrackx with CUDA 10.1 and ubuntu 20.04

Dear Jure:

I ran into problem running probtrackx_gpu with CUDA 10.1 in Ubuntu 20.4. Below is the command I used:

qunex_container dwi_probtrackx_dense_gpu \
  --sessionsfolder="${WORK_DIR}/${STUDY_NAME}/sessions" \
  --sessions="${SESSIONS}" \
  --omatrix3="yes" \
  --overwrite="yes" \
  --container="${QUNEX_CONTAINER}" \
  --bash_pre="module load CUDA/10.1" \
  --bash_post="export DEFAULT_CUDA_VERSION=10.1" \
  --bind="/usr/local/cuda-10.1/:/usr/local/cuda/" \
  --nv

And below is the error I got:

Running the following probtrackX GPU command:

---------------------------

   /opt/qunex/bash/qx_utilities/diffusion_tractography_dense/tractography_gpu_scripts/run_matrix3.sh /home/ehui/qunex/cimt/sessions V3 3000 no no

---------------------------

-- Queueing Probtrackx

PROBTRACKX2 VERSION GPU
Log directory is: /home/ehui/qunex/cimt/sessions/V3/hcp/V3/MNINonLinear/Results/Tractography
Running in seedmask mode
Loading tractography data
Number of Seeds: 192184
Dimensions Matrix3 91282 x 91282

Time Loading Data: 34 seconds


...................Allocated GPU 0...................
Device memory available (MB): 48418 ---- Total device memory(MB): 48685
Memory required for allocating data (MB): 826
CUDA Runtime Error: invalid device symbol

Any help would be greatly appreciated! Thanks so much!

Best,
Ed

Hm, never saw this before. Let me try this on our end and get back to you.

Jure

Ed, do you maybe have any older QuNex containers lying around (say 0.94.x or 0.95.x)? If so, could you run the above command with one of those. I am also having some issues with probtrackx in the latest container on our end.

Jure

Hi Jure:

I should have 0.94.14’s singularity in an external harddisk. Worst case, is there any way I could download 0.94.14’s singularity somewhere in your repository?

Many thanks!
Ed

Let me know if you need it and I will somehow get it to you.

Jure

Hi Jure:

Can you be so kind to share the singularity with me please?

Thanks very much!

Best,
Ed

Hi Ed,

I gave you access to the https://gitlab.qunex.yale.edu/qunexdev/qunexcontainer_dev repository. I am uploading 0.94.14 container in there (should be there in an hour or so). You can pull if from there the same way as from the public QuNex repository.

Jure

Hi Jure:

You are the best!

Thank so much and will let you know how it goes.

Best,
Ed

Hi Jure:

Unfortunately the same error occurred even when I used 0.94.14 container. Not sure if it had anything to do with the make of my GPU (which is A6000). I had no issues running the script in my another workstation which has a V100 nvidia GPU.

Please kindly advise.

Many thanks as always!
Ed

Hi,

I believe that A6000 supports only CUDA 11. Were you able to run bedpostx on A6000 with CUDA 10? That would be weird. We will be soon upgrading to FSL 6.0.6 and it seems like it supports CUDA 11 out of the box (Probtrackx GPU).

We plan to update to FSL 6.0.6 in the first half of January.

Jure

Hi Jure:

Yes I was able to run bedpostx with cuda 11 using my A6000;)

Many thanks!
Ed