[RESOLVED] Running hcp_diffusion with singularity

Issues
1

Dear QuNex Experts:

I was trying to use the singularity provided in gitlab to run hcp_diffusion with the following parameters:

PATH=${HOME}/qunex:${PATH}
export STUDY_NAME="quickstart006"
export WORK_DIR="${HOME}/qunex"
export QUNEX_CONTAINER="${WORK_DIR}/qunexcontainer/qunex_suite-0.93.6.sif" 
export RAW_DATA="${WORK_DIR}/data"
export INPUT_BATCH_FILE="${RAW_DATA}/006_batch.txt"
export INPUT_MAPPING_FILE="${RAW_DATA}/006_mapping.txt"
export SESSIONS="006"
export RUNTURNKEY_TYPE="local"
export RUNTURNKEY_STEPS="hcp_diffusion"

qunex_container hcp_diffusion   --rawdatainput="${RAW_DATA}"   --batchfile="${INPUT_BATCH_FILE}"   --mappingfile="${INPUT_MAPPING_FILE}"   --workingdir="${WORK_DIR}"   --projectname="${STUDY_NAME}"   --path="${WORK_DIR}/${STUDY_NAME}"   --sessions="${SESSIONS}"   --sessionids="${SESSIONS}"   --sessionsfoldername="sessions"  --nv --container="${QUNEX_CONTAINER}"

However, the qunex_container failed to run and returned the following errors:

........................ Running QuNex v0.93.6 ........................ 


--- Full QuNex call for command: hcp_diffusion 

gmri hcp_diffusion --rawdatainput="/home/ehui/qunex/data" --batchfile="/home/ehui/qunex/data/006_batch.txt" --mappingfile="/home/ehui/qunex/data/006_mapping.txt" --workingdir="/home/ehui/qunex" --projectname="quickstart006" --path="/home/ehui/qunex/quickstart006" --sessionsfoldername="sessions" --sessions="006" --sessionids="006" 

--------------------------------------------------------- 


WARNING: Parameter qx_cifti_tail was not specified. Its value was imputed from parameter hcp_cifti_tail and set to ''!
WARNING: Parameter qx_nifti_tail was not specified. Its value was imputed from parameter hcp_nifti_tail and set to ''!
WARNING: Parameter cifti_tail was not specified. Its value was imputed from parameter qx_cifti_tail and set to ''!
WARNING: Parameter nifti_tail was not specified. Its value was imputed from parameter qx_nifti_tail and set to ''!

# Generated by QuNex 0.93.6 on 2022-07-19_09.14.13.144751
#
=================================================================
gmri hcp_diffusion \
  --rawdatainput="/home/ehui/qunex/data" \
  --batchfile="/home/ehui/qunex/data/006_batch.txt" \
  --mappingfile="/home/ehui/qunex/data/006_mapping.txt" \
  --workingdir="/home/ehui/qunex" \
  --projectname="quickstart006" \
  --path="/home/ehui/qunex/quickstart006" \
  --sessionsfoldername="sessions" \
  --sessions="006" \
  --sessionids="006" \
=================================================================

Starting multiprocessing sessions in 006 with a pool of 1 concurrent processes

---- Running local

Starting processing of sessions 006 at Tuesday, 19. July 2022 09:14:13

------------------------------------------------------------
Session id: 006 
[started on Tuesday, 19. July 2022 09:14:13]
Running HCP DiffusionPreprocessing Pipeline [HCPStyleData] ...
ERROR: Unknown error occured: 
...................................
Traceback (most recent call last):
  File "/opt/qunex/python/qx_utilities/hcp/process_hcp.py", line 2161, in hcp_diffusion
    hcp = getHCPPaths(sinfo, options)
  File "/opt/qunex/python/qx_utilities/hcp/process_hcp.py", line 114, in getHCPPaths
    hcpbase                 = os.path.join(sinfo['hcp'], sinfo['id'] + options['hcp_suffix'])
KeyError: 'hcp'
...................................


HCP Diffusion Preprocessing completed on Tuesday, 19. July 2022 09:14:13
------------------------------------------------------------


===> Final report for command hcp_diffusion
... 006 ---> Error
===> Not all tasks completed fully!

Your help would be much appreciated! Thank you very much.

Best,
Ed

2

Hi Ed,

I see you are trying to update the run_turnkey QuickStart example to run hcp_diffusion. Unfortunately, hcp_diffusion is not currently supported with run_turnkey. Some of the parameters you are using above are used in run_turnkey but not in hcp_diffusion. We are planning on a major rework of run_turnkey which will make it easier to use and more flexible. For the time being you should run hcp_diffusion as a separate command.

A requirement for hcp_diffusion is a completed hcp_pre_freesurfer step. Here, you can use the QuickStart example to process your data up to hcp_pre_freesurfer. Next you should use a seperate hcp_diffusion call. How to set this up can be seen on the hcp_diffusion documentation page (https://qunex.readthedocs.io/en/latest/wiki/UsageDocs/HCPDiffusion.html). For general documentation regarding HCP Pipelines, consult https://qunex.readthedocs.io/en/latest/wiki/UsageDocs/HCPPreprocessing.html and https://qunex.readthedocs.io/en/latest/wiki/UsageDocs/RunningPreprocessHCP.html.

If the above answer did not work, let me know and I will help you out.

Jure

3

Hi Jure:

Once again, thanks so much for your detailed response indeed! I will also give your suggestions a try.

But interestingly, the following script currently runs fine even with hcp_diffusion in run_turnkey:

PATH=${HOME}/qunex:${PATH}
export STUDY_NAME="cimt"
export WORK_DIR="${HOME}/qunex"
export QUNEX_CONTAINER="${WORK_DIR}/qunexcontainer/qunex_suite-0.93.6.sif" export RAW_DATA="${WORK_DIR}/data"
export INPUT_BATCH_FILE="${RAW_DATA}/006_batch.txt"
export INPUT_MAPPING_FILE="${RAW_DATA}/006_mapping.txt"
export SESSIONS="001A,001B,001C,001D,002A,002B,002C,003A,003B,003C,003D,004A,004B,004C,006A,006C"
export RUNTURNKEY_TYPE="local"
export RUNTURNKEY_STEPS="create_study,map_raw_data,import_dicom,create_session_info,setup_hcp,create_batch,hcp_pre_freesurfer,hcp_freesurfer,hcp_post_freesurfer,hcp_fmri_volume,hcp_fmri_surface,hcp_diffusion"

qunex_container run_turnkey   --rawdatainput="${RAW_DATA}"   --batchfile="${INPUT_BATCH_FILE}"   --mappingfile="${INPUT_MAPPING_FILE}"   --workingdir="${WORK_DIR}"   --projectname="${STUDY_NAME}"   --path="${WORK_DIR}/${STUDY_NAME}"   --sessions="${SESSIONS}"   --sessionids="${SESSIONS}"   --sessionsfoldername="sessions"   --turnkeytype="${RUNTURNKEY_TYPE}"   --container="${QUNEX_CONTAINER}"  --turnkeysteps="${RUNTURNKEY_STEPS}" --overwrite="yes"

Thank so much again for your help!

Best,
Ed

4

My bad, basic runs of hcp_diffusion are indeed supported through run_turnkey, however if you need to tune additional parameters (see https://qunex.readthedocs.io/en/latest/wiki/UsageDocs/HCPDiffusion.html) you need to run the command directly.

Glad things are working now. If there are any other issues down the line, just ask here and I will help out!