[RESOLVED] Errors running hcp_msmall

Issues
1

Description:

Hi, I realize this question is similar to this one, but my error appears different. I successfully ran hcp_icafix, but I am having trouble with hcp_msmall finding the rest bold runs.

Call:

I ran these calls on Grace using qunex 0.98.3 and 0.98.4

qunex hcp_icafix \
	--batchfile="$my_study_folder/processing/batch_hcp.txt" \
	--sessionsfolder="$my_study_folder/sessions" \
	--hcp_icafix_bolds="GROUP_1:rest1,rest2|GROUP_2:task" \
	--hcp_matlab_mode="interpreted" \
	--container="$qunex_container"\
	--scheduler="SLURM,time=1-00:00:00,ntasks=1,cpus-per-task=2,mem-per-cpu=64000,partition=day,jobname=twogroup_icafix"

qunex hcp_msmall \
	--batchfile="$my_study_folder/processing/batch_hcp.txt" \
	--sessionsfolder="$my_study_folder/sessions" \
	--hcp_icafix_bolds="GROUP_1:rest1,rest2|GROUP_2:task" \
	--hcp_msmall_bolds="rest1,rest2" \
	--hcp_matlab_mode="interpreted" \
	--container="$qunex_container" \
	--scheduler="SLURM,time=1-00:00:00,ntasks=1,cpus-per-task=1,mem-per-cpu=32G,partition=day,jobname=msmall"

Logs:
example of a successful hcp_icafix log:

/gpfs/gibbs/pi/n3/Studies/Pittenger.OCD.SXprov/processing/logs/comlogs/done_hcp_icafix_GROUP_1_ta6499_2023-08-22_14.20.11.454164.log

example of an error log for hcp_msmall

/gpfs/gibbs/pi/n3/Studies/Pittenger.OCD.SXprov/processing/logs/batchlogs/qunex_container_2023-08-23_15.18.16.092718.txt

error message from the batchlog for this example:

# Generated by QuNex 0.97.3 on 2023-08-23_15.19.24.236347
#
=================================================================
gmri hcp_msmall \
  --sessionsfolder="/gpfs/gibbs/pi/n3/Studies//Pittenger.OCD.SXprov/sessions" \
  --hcp_icafix_bolds="GROUP_1:rest1,rest2|GROUP_2:task" \
  --hcp_msmall_bolds="rest1,rest2" \
  --hcp_matlab_mode="interpreted" \
  --sessions="/gpfs/gibbs/pi/n3/Studies//Pittenger.OCD.SXprov/processing/batch_hcp.txt" \
  --sessionids="ta6499" \
=================================================================

Starting multiprocessing sessions in /gpfs/gibbs/pi/n3/Studies//Pittenger.OCD.SXprov/processing/batch_hcp.txt with a pool of 1 concurrent processes

---- Running local

Starting processing of sessions ta6499 at Wednesday, 23. August 2023 15:19:24

------------------------------------------------------------
Session id: ta6499 
[started on Wednesday, 23. August 2023 15:19:24]
Running HCP MSMAll pipeline [HCPStyleData] ...

Comparing bolds with those specifed via parameters
     ... all bolds specified via hcp_icafix_bolds are present

Multi-run HCP ICAFix on 2 groups
---> WARNING: multiple groups provided in hcp_icafix_bolds, running MSMAll by using only the first one [GROUP_1]!
---> ERROR: bold (3, 'bold3', 'rest2', {'ima': '171', 'se': '1', 'name': 'bold3', 'task': 'rest2', 'ext': 'HCP_bold_2.5mm_.7TR_MB6_Rest_Post'}) %s used in hcp_msmall_bolds but not found in hcp_icafix_bolds!
---> ERROR: bold (3, 'bold3', 'rest2', {'ima': '171', 'se': '1', 'name': 'bold3', 'task': 'rest2', 'ext': 'HCP_bold_2.5mm_.7TR_MB6_Rest_Post'}) %s used in hcp_msmall_bolds but not found in hcp_icafix_bolds!

ERROR in completing hcp_msmall:
     ... invalid input parameters!
2

Hi Rachel, I do not think you can run hcp_msmall over GROUPS that do not include hcp_msmall_bolds. I believe the correct processing for this would be:

qunex hcp_icafix \
	--batchfile="$my_study_folder/processing/batch_hcp.txt" \
	--sessionsfolder="$my_study_folder/sessions" \
	--hcp_icafix_bolds="GROUP_1:rest1,rest2,task" \
	--hcp_matlab_mode="interpreted" \
	--container="$qunex_container"\
	--scheduler="SLURM,time=1-00:00:00,ntasks=1,cpus-per-task=2,mem-per-cpu=64000,partition=day,jobname=twogroup_icafix"

qunex hcp_msmall \
	--batchfile="$my_study_folder/processing/batch_hcp.txt" \
	--sessionsfolder="$my_study_folder/sessions" \
	--hcp_icafix_bolds="GROUP_1:rest1,rest2,task" \
	--hcp_msmall_bolds="rest1,rest2" \
	--hcp_matlab_mode="interpreted" \
	--container="$qunex_container" \
	--scheduler="SLURM,time=1-00:00:00,ntasks=1,cpus-per-task=1,mem-per-cpu=32G,partition=day,jobname=msmall"

If I am not mistaken, this is the most similar to how HCP released data is processed.

3

Hi Jure, thanks for your suggestion. I should have included that I also tried it that way, but the (nearly) same error is thrown:

error log:

/gpfs/gibbs/pi/n3/Studies/Pittenger.OCD.SXprov/processing/logs/batchlogs/SLURM_hcp_msmall_job51.2023-08-29_13.51.58.206005.log

# Generated by QuNex 0.98.5 on 2023-08-29_13.52.02.430428#
=================================================================
gmri hcp_msmall \
  --sessionsfolder="/gpfs/gibbs/pi/n3/Studies//Pittenger.OCD.SXprov/sessions" \
  --hcp_icafix_bolds="GROUP_1:rest1,rest2,task" \
  --hcp_msmall_bolds="rest1,rest2" \
  --hcp_matlab_mode="interpreted" \
  --container="/gpfs/gibbs/pi/n3/software/Singularity/qunex_suite-0.98.4.sif" \
  --sessions="/gpfs/gibbs/pi/n3/Studies//Pittenger.OCD.SXprov/processing/batch_hcp.txt" \
  --sessionids="ta7806" \
=================================================================

Starting multiprocessing sessions in /gpfs/gibbs/pi/n3/Studies//Pittenger.OCD.SXprov/processing/batch_hcp.txt with a pool of 1 concurrent processes

---- Running local

Starting processing of sessions ta7806 at Tuesday, 29. August 2023 13:52:03

------------------------------------------------------------
Session id: ta7806 
[started on Tuesday, 29. August 2023 13:52:03]
Running HCP MSMAll pipeline [HCPStyleData] ...

Comparing bolds with those specifed via parameters
     ... all bolds specified via hcp_icafix_bolds are present

Multi-run HCP ICAFix on 1 groups
---> ERROR: bold (3, 'bold3', 'rest2', {'ima': '171', 'se': '1', 'name': 'bold3', 'task': 'rest2', 'ext': 'HCP_bold_2.5mm_.7TR_MB6_Rest_Post'}) %s used in hcp_msmall_bolds but not found in hcp_icafix_bolds!
---> ERROR: bold (3, 'bold3', 'rest2', {'ima': '171', 'se': '1', 'name': 'bold3', 'task': 'rest2', 'ext': 'HCP_bold_2.5mm_.7TR_MB6_Rest_Post'}) %s used in hcp_msmall_bolds but not found in hcp_icafix_bolds!

ERROR in completing hcp_msmall:
     ... invalid input parameters!


HCP MSMAll completed on Tuesday, 29. August 2023 13:52:03
------------------------------------------------------------


===> Final report for command hcp_msmall
... ta7806 ---> HCP MSMAll failed
===> Success status not reported for some or all tasks

Any thoughts on what I should try next?

4

Hi, to run QuNex through the container, you need to use qunex_container instead of qunex. First, let us try:

# prepare variables
my_study_folder="/gpfs/gibbs/pi/n3/Studies//Pittenger.OCD.SXprov"
qx_container="/gpfs/gibbs/pi/n3/software/Singularity/qunex_suite-0.98.5.sif"

# run qunex via the container
qunex_container hcp_msmall \
	--sessionsfolder="$my_study_folder/sessions" \
	--batchfile="$my_study_folder/processing/batch_hcp.txt" \
	--hcp_icafix_bolds="GROUP_1:rest1,rest2,task" \
	--hcp_msmall_bolds="rest1,rest2" \
	--container="$qx_container" \
	--scheduler="SLURM,time=1-00:00:00,ntasks=1,cpus-per-task=1,mem-per-cpu=32G,partition=day,jobname=msmall"

If this does not work, I will check what is going on on Grace. I believe I have access to the study folder.

5

Thanks --unfortunately, it’s still failing with the identical error when I run it using qunex_container (and using that version of qx_container). Here’s a sample log again:

/gpfs/gibbs/pi/n3/Studies/Pittenger.OCD.SXprov/processing/logs/batchlogs/SLURM_hcp_msmall_job51.2023-08-29_13.51.58.206005.log

6

If I see correctly, the --hcp_icafix_bolds parameter was set to --hcp_icafix_bolds="GROUP_1:rest1,rest2|GROUP_2:task" in hcp_icafix? This parameter needs to match between hcp_icafix and hcp_msmall. So, try rerunning:

# prepare variables
my_study_folder="/gpfs/gibbs/pi/n3/Studies//Pittenger.OCD.SXprov"
qx_container="/gpfs/gibbs/pi/n3/software/Singularity/qunex_suite-0.98.5.sif"

# icafix
qunex_container hcp_icafix \
	--sessionsfolder="$my_study_folder/sessions" \
	--batchfile="$my_study_folder/processing/batch_hcp.txt" \
	--hcp_icafix_bolds="GROUP_1:rest1,rest2,task" \
	--overwrite="yes" \
	--container="$qunex_container" \
	--scheduler="SLURM,time=1-00:00:00,ntasks=1,cpus-per-task=2,mem-per-cpu=64000,partition=day,jobname=icafix"

# msmall
qunex_container hcp_msmall \
	--sessionsfolder="$my_study_folder/sessions" \
	--batchfile="$my_study_folder/processing/batch_hcp.txt" \
	--hcp_icafix_bolds="GROUP_1:rest1,rest2,task" \
	--hcp_msmall_bolds="rest1,rest2" \
	--overwrite="yes" \
	--container="$qx_container" \
	--scheduler="SLURM,time=1-00:00:00,ntasks=1,cpus-per-task=1,mem-per-cpu=32G,partition=day,jobname=msmall"
7

Hi Jure – sorry, maybe I’m a little confused – the intention is to run hcp_icafix separately on rest vs. task, hence the two groups:

--hcp_icafix_bolds="GROUP_1:rest1,rest2|GROUP_2:task"

Are you saying I should rerun both hcp_icafix and then subsequently hcp_msmall with the "GROUP_2" setting removed? (e.g., changing:

--hcp_icafix_bolds="GROUP_1:rest1,rest2,|GROUP_2:task"

to

--hcp_icafix_bolds="GROUP_1:rest1,rest2,task"

in both commands?

Just for clarity: One goal of these steps is to identify the rest-specific ICA-derived networks in our data, so I want to avoid lumping in the task-based runs at this stage.

8

I believe the “default” procedure is to lump everything together in ICAFix and then separate in MSMAll, hence

--hcp_icafix_bolds="GROUP_1:rest1,rest2,task"

In both commands and then

--hcp_msmall_bolds="rest1,rest2" \

in hcp_msmall command.

If I am not mistaken this is how the publicly released HCP datasets were processed. Maybe best to ask this on HCP users group. hcp_msmall does not work if hcp_icafix_bolds parameter does no specify the same group and components/bolds as the previous hcp_icafix and if hcp_msmall_bolds parameter is not a subset of hcp_icafix_bolds.

What most users do is that they do not set hcp_icafix_bolds manually and use the default, which will take all bolds in the session and then specify rest bolds in hcp_msmall_bolds.