[RESOLVED] Issue running run_qc

Issues
1

Description
I’m trying to generate DWI qc files and it seems there is a problem with parsing the modality of the command. The commands throw this error in the terminal:

---> Using  individual information files. Verifying that  exists.
 /gpfs/project/fas/n3/Studies/Connectome/subjects//101309/ NOT found. Check BOLD inputs.

This is strange because it shouldn’t be checking for BOLD input at all and it seems the command is passing some sort of empty string. I also ran it outside of the container on a node, where the same error persists.

Call
The following commands were run from /gpfs/project/fas/n3/Studies/Connectome/processing/logs/qc after sourcing the qunex environment

qunexContainer run_qc \
--sessionsfolder="/gpfs/project/fas/n3/Studies/Connectome/subjects/" \
--sessions="101309" \
--outpath='/gpfs/project/fas/n3/Studies/Connectome/subjects/QC/DWI' \
--templatefolder='/gpfs/project/fas/n3/software/qunex/qx_library/data/scenes/qc' \
--modality='DWI' \
--scenezip \
--dwilegacy='no' \
--dtifitqc='yes' \
--dwidata='data' \
--dwipath='Diffusion' \
--overwrite='no' \
--container="/gpfs/project/fas/n3/software/Singularity/qunex_suite-latest.sif"

qunexContainer run_qc \
--sessionsfolder="/gpfs/project/fas/n3/Studies/Connectome/subjects/" \
--sessions="101309" \
--outpath='/gpfs/project/fas/n3/Studies/Connectome/subjects/QC/DWI' \
--templatefolder='/gpfs/project/fas/n3/software/qunex/qx_library/data/scenes/qc' \
--modality='DWI' \
--scenezip \
--dwilegacy='no' \
--bedpostxqc='yes' \
--dwidata='data' \
--dwipath='Diffusion' \
--overwrite='no' \
--container="/gpfs/project/fas/n3/software/Singularity/qunex_suite-latest.sif"

Logs

Here is a sample log output from the two commands:
dtifitqc:
/gpfs/project/fas/n3/Studies/Connectome/processing/logs/comlogs/error_run_qc_101309_2021-07-22_13.43.0063239709.log

bedpostx qc:
/gpfs/project/fas/n3/Studies/Connectome/processing/logs/comlogs/error_run_qc_101309_2021-07-22_13.45.0306674279.log

2

Hi Amber,

Thanks for reporting this. I will test everything and reply here once I find out what is going on.

3

New version was just released on grace, the commands below worked fine for me:

qunex run_qc \
--sessionsfolder="/gpfs/project/fas/n3/Studies/Connectome/subjects" \
--sessions="101309" \
--outpath='/gpfs/project/fas/n3/Studies/Connectome/subjects/QC/DWI' \
--modality='DWI' \
--scenezip \
--dwilegacy='no' \
--dtifitqc='yes' \
--dwidata='data' \
--dwipath='Diffusion' \
--overwrite='no'

qunex run_qc \
--sessionsfolder="/gpfs/project/fas/n3/Studies/Connectome/subjects/" \
--sessions="101309" \
--outpath='/gpfs/project/fas/n3/Studies/Connectome/subjects/QC/DWI' \
--modality='DWI' \
--scenezip \
--dwilegacy='no' \
--bedpostxqc='yes' \
--dwidata='data' \
--dwipath='Diffusion' \
--overwrite='no'
4

Hi Jure, I tried this command on the bigmem node from /gpfs/project/fas/n3/Studies/Connectome/processing/logs/qc (Command 1). I then tried the same command with the container (Command 2) and then with the container and the scheduler (Command 3). The command runs for command 1 and 3. There does seem to be an overwrite bug (more details in the command 1 section) and I get a strange error when just running the command with the container on a node (see command 2 for details).

Command 1: Runs correctly but there does seem to be an overwrite bug (if overwrite set to true it deletes all of the DWI QC files for the subject, even if only 1 type of DWI file is set to overwrite).
qunex run_qc
–sessionsfolder=“/gpfs/project/fas/n3/Studies/Connectome/subjects”
–sessions=“101309”
–outpath=‘/gpfs/project/fas/n3/Studies/Connectome/subjects/QC/DWI’
–modality=‘DWI’
–scenezip
–dwilegacy=‘no’
–dtifitqc=‘yes’
–bedpostxqc=‘yes’
–dwidata=‘data’
–dwipath=‘Diffusion’
–overwrite=‘yes’

I ran with with bedpostxqc=‘yes’ and dtiqc=‘yes’. Then I ran it again with bedpostxqc=‘yes’ and dtiqc=‘no’. It seems there is a bug such that when overwrite it set to “yes” it will delete all of the subjects QC data (bedpostx, dtifi) even if you’re rerunning only one set of data (bedpostx). Other than that this command worked for me.

Command 2 Running command with the latest container results in an error in the terminal.
Then I tried to run it through the container like so:

qunex_container run_qc \
--sessionsfolder="/gpfs/project/fas/n3/Studies/Connectome/subjects" \
--sessions="101309" \
--outpath='/gpfs/project/fas/n3/Studies/Connectome/subjects/QC/DWI' \
--modality='DWI' \
--scenezip \
--dwilegacy='no' \
--dtifitqc='yes' \
--bedpostxqc='yes' \
--dwidata='data' \
--dwipath='Diffusion' \
--overwrite='yes' \
--container="/gpfs/project/fas/n3/software/Singularity/qunex_suite-0.90.10.sif"

And I got the following error in terminal:

qunex_container run_qc
–sessionsfolder=“/gpfs/project/fas/n3/Studies/Connectome/subjects”
–sessions=“101309”
–outpath=‘/gpfs/project/fas/n3/Studies/Connectome/subjects/QC/DWI’
–modality=‘DWI’
–scenezip
–dwilegacy=‘no’
–dtifitqc=‘yes’
–bedpostxqc=‘yes’
–dwidata=‘data’
–dwipath=‘Diffusion’
–overwrite=‘yes’
–container=“/gpfs/project/fas/n3/software/Singularity/qunex_suite-0.90.10.sif”

→ QuNex will run the command over 1 sessions. It will utilize:

Scheduled jobs: 1 
Maximum sessions run in parallel for a job: 1.
Maximum elements run in parallel for a session: 1.
Up to 1 processes will be utilized for a job.

Job #1 will run sessions: 101309

Traceback (most recent call last):
File “/gpfs/project/fas/n3/software/qunex/bin/qunex_container”, line 964, in
main()
File “/gpfs/project/fas/n3/software/qunex/bin/qunex_container”, line 961, in main
subprocess.Popen(containerCommand, shell=True, parsessions=parsessions)
TypeError: init() got an unexpected keyword argument ‘parsessions’

Command 3 Running command with the latest container with the scheduler works fine. In the log file I do notice some permissions errors but it seems like all of the qc files were generated appropriately so I’m not sure how problematic those warnings are.

qunex_container run_qc \
--sessionsfolder="/gpfs/project/fas/n3/Studies/Connectome/subjects" \
--sessions="101309" \
--outpath='/gpfs/project/fas/n3/Studies/Connectome/subjects/QC/DWI' \
--modality='DWI' \
--scenezip \
--dwilegacy='no' \
--dtifitqc='yes' \
--bedpostxqc='yes' \
--dwidata='data' \
--dwipath='Diffusion' \
--overwrite='yes' \
--scheduler="SLURM,time=1-00:00:00,ntasks=4,cpus-per-task=1,mem-per-cpu=20000,partition=pi_anticevic" \
--container="/gpfs/project/fas/n3/software/Singularity/qunex_suite-0.90.10.sif"

log file located here:
/gpfs/project/fas/n3/Studies/Connectome/processing/logs/comlogs/done_run_qc_101309_2021-07-23_15.18.0017773151.log

5

The conainer is not yet updated, it will be early next week, this is probably the reason for error #2. I recorded the remaining bugs and will fix them, but it might take some time until they are propagated into a release. They are not as crucial since there is workaround to avoid them (rerun the qc step that gets deleted).

Jure

6

The issue has been resolved. The bug fix will be released in the next version of QuNex.

1 Like