[RESOLVED] Running over multiple sessions

1

Dear Jure/Lining:

Forgive me for asking this trivial question.

I was trying to run qunex_container over multiple sessions using the following command:

export SDIR="/hdd1"
PATH=${SDIR}/qunex:${PATH}
export STUDY_NAME="cimt"
export WORK_DIR="${SDIR}/qunex"
export QUNEX_CONTAINER="${SDIR}/qunex/qunexcontainer/qunex_suite-0.96.2a.sif" 
export QUNEX_CONTAINER="${SDIR}/qunex/qunexcontainer/qunex_suite-0.99.2d.sif" 
export RAW_DATA="${WORK_DIR}/data"
export INPUT_BATCH_FILE="${RAW_DATA}/006_batch.txt"
export INPUT_MAPPING_FILE="${RAW_DATA}/006_mapping__.txt"
export SESSIONS="001A,019A" #,010A,009C,010C" 
export RUNTURNKEY_TYPE="local"
export RUNTURNKEY_STEPS="create_study,map_raw_data,import_dicom,create_session_info,setup_hcp,create_batch,hcp_pre_freesurfer,hcp_freesurfer,hcp_post_freesurfer,hcp_fmri_volume,hcp_fmri_surface,hcp_diffusion,map_hcp_data,create_bold_brain_masks,compute_bold_stats,create_stats_report,extract_nuisance_signal,preprocess_bold"
export RUNTURNKEY_STEPS="create_study,map_raw_data,import_dicom,create_session_info,setup_hcp,create_batch"

rm "${WORK_DIR}/${STUDY_NAME}/sessions/specs/$(basename ${INPUT_MAPPING_FILE})"

qunex_container run_turnkey   \
        --rawdatainput="${RAW_DATA}"   \
        --paramfile="${INPUT_BATCH_FILE}"   \
        --mappingfile="${INPUT_MAPPING_FILE}"   \
        --workingdir="${WORK_DIR}"   \
        --projectname="${STUDY_NAME}"   \
        --path="${WORK_DIR}/${STUDY_NAME}"   \
        --sessions="${SESSIONS}"   \
        --sessionids="${SESSIONS}"   \
        --sessionsfoldername="sessions"   \
        --bind="/hdd1:/hdd1" \
        --turnkeytype="${RUNTURNKEY_TYPE}"   \
        --nv \
        --container="${QUNEX_CONTAINER}"  \
        --turnkeysteps="${RUNTURNKEY_STEPS}" \
        --overwrite="yes"

But for some reason, the script could not continue and was stuck, and instead of separate folder for each session, there was a single folder with the name 001A,019A.

Can you be so kind as to tell me the right way to run the script over multiple sessions please?

Many thanks as always,
Ed

2

Hi Ed,

This is weird as your call looks OK. I have never encountered this, it seems like for some reason QuNex is not splitting 001A,019A into 001A and 019A and tries to use them as a single session with a comma in between. It seems like a run_turnkey issue. run_turnkey has some issues and we are in the process of deprecating it and replacing it with a new engine called run_recipe. We are planning to release the next QuNex version within a couple of weeks and it will have run_recipe in it. It is a much more flexible and powerful end-to-end processing engine in comparison with run_turnkey. When we do use run_turnkey across several sessions ourselves, we usually trigger the command run for each session separately. Above you would thus have 2 qunex_container run_turnkey executions, one for the first session and one for the second session. Because of the previously mentioned issues we rarely use run_turnkey but most often default to execution of individual commands in a chain. Like I said, we are releasing a new engine that fixes all this soon.

If you still wish to proceed with run_turnkey across multiple sessions simultaneously I made a small fix in the qunex_container that should fix the issue you are seeing. Try replacing your qunex_container script with this one: qunex_container (42.9 KB) and also consider adding --parssessions=2 to the call so processing commands will be execute over both sessions in paralell.

Hope this helped.

Best, Jure

3

Hi Jure:

Many thanks for your reply indeed. I am glad to know there will be a new release that can fix this issue.

I just tried the new container but it is still not working as intended. Below is the output

image
image774×964 47.7 KB

Note that instead of two separate folder for each session, there was only one folder called “2” and underneath it, the “inbox” contains all the zip file stored under the data folder.

Can you be so kind to see if this could be fixed? Many thanks!

Best,
Ed

4

The problem is that I am unable to reproduce your behavior. I tried running everything in a similar fashion as you did over 2 sessions and it worked on my end. In my previous fixed I just tried to make session parsing a bit more robust in the qunex_container script. An easy fix right now is to either:

  1. Execute run_turnkey for each session independently. So, you set export SESSIONS="001A" run the command and then set export SESSIONS="019A" and run it again. You can run the two in parallel in two open terminals separately.

  2. Do not use run_turnkey but execute everything in a command-by-command basis. You can find a tutorial for this at QuNex quick start using a Docker container — QuNex documentation in the section titled Option B: Step-by-step execution. This is our go-to approach and works much more robustly then run_turnkey.

Best, Jure

5

Hi Jure:

Understood, thanks so much for your detailed suggestions indeed!

Much appreciate it!

Best,
Ed