[RESOLVED] Distance correction for probtrackx (-pd and -ompl flags)

Hello, I’ve been running probtrackx using QuNex and it seems there are some options that are available for probtrackx using FSL’s FDT but I don’t see that these options are implementable through QuNex. Specifically, there are two flags I would like to run probtrackx with : -pd for generated distance-corrected connectivity matrices and the --ompl for saving the streamline length distance matrix. I don’t see how to set these flags with the current QuNex implementation. Is there are easy way to implement this on my end or would this have to be built into the QuNex implementation?

qunex dwi_probtrackx_dense_gpu \
    --sessionsfolder="<path_to_study_sessions_folder>" \
    --sessions="<comma_separated_list_of_cases>" \
    --omatrix1="<matrix1_model>" \
    --omatrix3="<matrix3_model>" \
    --nsamplesmatrix1="<number of samples for Matrix1>" \
    --nsamplesmatrix3="<number of samples for Matrix3>" \
    --overwrite="no" \
    --scheduler="<name_of_cluster_scheduler_and_options>"

Hi Amber,

Thanks for this suggestion. This should be easy to implement, I will do my best so it makes it into the next release (no eta for this yet though). If you urgently need this, then you can execute a qunex dwi_probtrackx_dense_gpu without these parameters and check out the outgoing call to FSL’s function in the log. Next, you can amend this call with the options that are currently missing. So you would execute FSL’s FDT directly.

Jure

We added two additional parameters to the dwi_probtrackx_dense_gpu command. With --distancecorrection="yes" and --storestreamlineslength="yes" you will be able to get the desired behavior. The upgrade will be pushed into the next QuNex release.

1 Like

Hi Jure, I ran the command and it runs but I encounter an out-of-memory error. QuNex correctly reports that the step has passed, when is should report that the step failed. I only ran a subject subject and the command seemed to run fine without the distance flags, so I’m also not sure why I’m getting the memory error. Maybe I’m not optimizing the slurm command very well?

Command:
qunex_container dwi_probtrackx_dense_gpu
–sessionsfolder="/gpfs/project/fas/n3/Studies/Connectome/subjects"
–sessions=“100206”
–omatrix1=‘yes’
–omatrix3=‘yes’
–distancecorrection=“yes”
–storestreamlineslength=“yes”
–nv
–overwrite=“yes”
–parsessions=1
–scheduler=“SLURM,time=12:00:00,ntasks=1,cpus-per-task=1,mem-per-cpu=64000,partition=pi_anticevic_gpu,gres=gpu:1,jobname=ProbtrackxGPUDense”
–container="/gpfs/project/fas/n3/software/Singularity/qunex_suite-0.91.3.sif"

log:
/gpfs/project/fas/n3/Studies/Connectome/processing/logs/probtrackx/slurm-32777984.out

Additionally, I noticed that for the recent subjects I have rerun it seems that the Mat1.dconn.nii and the Mat1_transp.dconn.nii are left in the folder and are unzipped. These files aren’t present in the tractography folders in subjects who haven not been rerun recently. An example subject that was run a long time ago and the doesn’t have these large files is 110411 (tractography folder: /gpfs/project/fas/n3/Studies/Connectome/subjects/110411/hcp/110411/MNINonLinear/Results/Tractography). I’m wondering if maybe a step isn’t finishing the Mat1 post processing script. If not, I was hoping the commands could be updated such that these very large files are zipped so they don’t take up so much space.

The problem when QuNex says that the process finished successfully when it indeed did not is a tricky one. QuNex often calls scripts and programs from other developers/researchers and here we rely on how they report their results. Some scripts report a success (exit code 0) even when things failed. In some cases we put additional safety checks in place (like checking for generated files) on top of relying on reports from others, but this is not possible in all cases (e.g. when a script only modifies a file and does not create it). I will check if we could have a more robust check in the case of dwi_probtrackx_dense_gpu.

As for the out-of-memory issue, it is exactly what it says. You did not reserve enough memory for running. There are special nodes on grace that have a lot of memory (128GB+), you should schedule your run on a big memory node that also has a GPU. I have never done this, so I cannot really help you out here. You should consult the IT team that takes care of grace.

I will check what is going on with the post-processing step. Could this stay unzipped because of the out-of-memory issues, or are you seeing this in runs that do not succeed?

Cheers, Jure

Okay, I will check in with YCRC regarding nodes with more memory and a GPU and I’ll send an update.

Re: unzipped files/post-processing step : it appears that these files are generated when the run is successful. One example log file is /gpfs/project/fas/n3/Studies/Connectome/processing/logs/probtrackx/old/slurm-18745722.out. Probtrackx was run successfully, there doesn’t seem to be a memory error, and the Mat1.dconn.nii.gz and Mat1_transp.dconn.nii.gz are in this subject’s tractography folder (/gpfs/project/fas/n3/Studies/Connectome/subjects/121416/hcp/121416/MNINonLinear/Results/Tractography).

Ah, then this is not really a bug. The file zipping is the last part of the command and if the command crashes before the file zipping part or during the file zipping part then zipping will not be concluded.

Regarding the zipping of the files - I don’t think the code is crashing. These files are present for the ~97 subjects I have recently rerun, and all of these subjects report no memory issues during processing. I do think you’re right that it’s not a bug, but I think zipping that files would be a nice feature to save space. I propose a code modification to the post-processing script to zip these dconn files.

I believe this is the location of the script on Grace: /gpfs/project/fas/n3/software/qunex/bash/qx_utilities/diffusion_tractography_dense/tractography_gpu_scripts/post_proc_matrix1.sh. It seems the the Conn3 and Conn1 post-processing have slightly different steps and Conn1 generates these two additional dconn files. I also see that the code doesn’t zip these dconn files for Conn1. I propose adding the following lines to the end of the post_proc_matrix1.sh script:

gzip --force $ResultsFolder/Mat1.dconn.nii --fast
gzip --force $ResultsFolder/Mat1_transp.dconn.nii--fast

Hi Amber,

thanks for investigating this in great detail and providing a concrete proposal for the upgrade. I will upgrade the develop version today. The upgrade will then make it into the next released version (both source and container). If this is urgent for the processing you are working on, I can hotfix the current version (0.91.4). This way, the updated version will be available tomorrow or the day after. Let me know if this is important for you.

Jure

One additional question. Are the Mat1.dconn.nii Mat1_transp.dconn.nii always generated or only when probtracx is ran with certain parameters? Thanks.

I think they are always created - it’s my understanding that Probtrackx generates fdt_matrix1.dot. This is converted to Mat1.dconn.nii which is then transposed in Mat1_transp.dconn.nii through qunex code using workbench. The Conn1.dconn.nii file is then created by averaging Mat1.dconn.nii and Mat1_transp.dconn.nii, so the final Conn1.dconn.nii matrix is symmetrized. In this case, I’m unsure that the Mat1.dconn.nii and Mat1_transp.dconn.nii files are even worth keeping. It appears these files must have been deleted (or archived) at some point for subjects that were run before I started working with the data.

This is also not urgent - the files are taking up space but if the space becomes an issue, which is unlikely, I can zip them myself - no need for a hotfix. Thanks Jure!

Hi Jure - I got the probtrackx command to work with distance correction by increasing the memory from 62.5 to 100gb. I did self on the slurm job and it required 68gb to run. It seems there are two additional files outputted when “storestreamlineslength” is set to yes. I propose adding a feature such that the post-processing commands also convert and zip these additional files if they exist in the subject’s tractography directory. I pasted proposed code for this below:

Append to post_proc_matrix1.sh (after “OutFileTemp” is defined) :

if [ -f ${ResultsFolder}/fdt_matrix1_lengths.dot ]; then
${Caret7_command} -probtrackx-dot-convert ${ResultsFolder}/fdt_matrix1_lengths.dot ${ResultsFolder}/Mat1_lengths.dconn.nii -row-cifti ${TemplateFolder}/91282_Greyordinates.dscalar.nii COLUMN -col-cifti ${TemplateFolder}/91282_Greyordinates.dscalar.nii COLUMN

${Caret7_command} -cifti-transpose ${ResultsFolder}/Mat1_lengths.dconn.nii ${ResultsFolder}/Mat1_lengths_transp.dconn.nii

${Caret7_command} -cifti-average ${ResultsFolder}/${OutFileTemp}_lengths.dconn.nii -cifti ${ResultsFolder}/Mat1_lengths.dconn.nii -cifti ${ResultsFolder}/Mat1_lengths_transp.dconn.nii

gzip --force $ResultsFolder/Mat1_lengths.dconn.nii --fast
gzip --force $ResultsFolder/Mat1_lengths_transp.dconn.nii–fast
gzip --force $ResultsFolder/${OutFileTemp}_lengths.dconn.nii --fast
fi

Append to post_proc_matrix3.sh after “OutFileTemp” is defined :

if [ -f ${ResultsFolder}/fdt_matrix3_lengths.dot ]; then
${Caret7_command} -probtrackx-dot-convert ${ResultsFolder}/fdt_matrix3_lengths.dot ${ResultsFolder}/${OutFileTemp}_lengths.dconn.nii -row-cifti ${TemplateFolder}/91282_Greyordinates.dscalar.nii COLUMN -col-cifti ${TemplateFolder}/91282_Greyordinates.dscalar.nii COLUMN -make-symmetric
gzip --force $ResultsFolder/${OutFileTemp}_lengths.dconn.nii --fast
fi

Thanks for providing this. Some more pressing bugs appeared in the current release, so I will hotfix this today or tomorrow. 0.91.5 should be available on grace soon.

1 Like

Okay, I’ve tested out the new container and there is a bug preventing the mat3_lengths.dot file being converted to Conn3_lengths.dconn.nii file. From the error it seems that template mapping file isn’t a compatible size (which is a little confusing since it has the same dimensions as the mat3.dot file).

command

qunex_container dwi_probtrackx_dense_gpu
–sessionsfolder=“/gpfs/project/fas/n3/Studies/Connectome/subjects”
–sessions=“100206,100610,101006,101309,101915,102311,102513,103414,103515,103818,105115,106016”
–omatrix1=‘yes’
–omatrix3=‘yes’
–distancecorrection=‘yes’
–storestreamlineslength=‘yes’
–nv
–overwrite=“no”
–parsessions=1
–scheduler=“SLURM,time=12:00:00,ntasks=1,cpus-per-task=1,mem-per-cpu=70000,partition=pi_anticevic_gpu,gres=gpu:1,jobname=ProbtrackxGPUDense”
–container=“/gpfs/project/fas/n3/software/Singularity/qunex_suite-0.91.6.sif”

log
Here’s an example log file: /gpfs/project/fas/n3/Studies/Connectome/processing/logs/probtrackx/slurm-33454927.out

and here’s the error I see:

While running:

/opt/workbench/workbench-1.5.0/bin_rh_linux64/…/exe_rh_linux64/wb_command -probtrackx-dot-convert /gpfs/project/fas/n3/Studies/Connectome/subjects/105115/hcp/105115/MNINonLinear/Results/Tractography/fdt_matrix3_lengths.dot /gpfs/project/fas/n3/Studies/Connectome/subjects/105115/hcp/105115/MNINonLinear/Results/Tractography/Conn3_lengths.dconn.nii -row-cifti /opt/qunex/qx_library/etc/diffusion_tractography_dense/templates/91282_Greyordinates.dscalar.nii COLUMN -col-cifti /opt/qunex/qx_library/etc/diffusion_tractography_dense/templates/91282_Greyordinates.dscalar.nii COLUMN -make-symmetric

ERROR: dimensions line in .dot file doesn’t agree with provided row/column spaces

Hi Amber, I assume that this appens only when --omatrix3="yes", --distancecorrection="yes" and --storestreamlineslength="yes"? The runs without distancecorrection and storestreamlineslength worked fine?

Hi Jure - I think the error only occurs when --storestreamlineslength=“yes” and --omatrix3=“yes”.

Thanks, I identified the problematic line of code:

${Caret7_command} -probtrackx-dot-convert ${ResultsFolder}/fdt_matrix3_lengths.dot ${ResultsFolder}/${OutFileTemp}_lengths.dconn.nii -row-cifti ${TemplateFolder}/91282_Greyordinates.dscalar.nii COLUMN -col-cifti ${TemplateFolder}/91282_Greyordinates.dscalar.nii COLUMN -make-symmetric

Now, I need to figure out what is not OK. If you have any ideas, they are more than welcome.

Hi Jure - I wonder if there is an issue earlier in the processing that causes the fdt_matrix3_lengths.dot file to be truncated for matrix3? Is fdt_matrix3_lengths.dot the automatic output of FSL? The length of the fdt_matrix1_lengths.dot and the fdt_matrix1.dot files seem to be the same for each subject (but different across subjects). However, the lengths of the fdt_matrix3.dot file and the fit_matrix3_lengths.dot files are not the same for each subject. I’m not sure if this is actually important since I don’t know much about these dot files or how they are generated. I do think there is something wrong with the dot file itself, as opposed to converting the dot file, since the code to convert works just fine for the fdt_matrix3.dot file.

cd /gpfs/project/fas/n3/Studies/Connectome/subjects
for dir in [1-9]*/hcp/*/MNINonLinear/Results/Tractography/; do wc -l $dir/fdt_matrix1_lengths.dot; wc -l $dir/fdt_matrix1.dot; echo ""; done # same length for each subject 
for dir in [1-9]*/hcp/*/MNINonLinear/Results/Tractography/; do wc -l $dir/fdt_matrix3_lengths.dot; wc -l $dir/fdt_matrix3.dot; echo ""; done # different lengths for each subjects

Yes, the fdt_matrix3_lengths.dot is the automatic FSL’s output when the --storestreamlineslength="yes" option is used and I have absolutely no control over it. The only “solution” that currently comes to my mind is removing the probtrackx-dot-convert call for fdt_matrix3_lengths.dot and just zipping the output (.dot file) directly. If you have a better solution, please let me know.

Another thing worth checking out is if the Matrix3 execution actually completed successfully, it could be that the process crashed because it ran out of memory or something similar, and it is because of this that the output gets truncated. I will start a run on a single subject today and see where that gets me.