[RESOLVED] Running map_raw_data in parallel

Use Questions
1

Hi there,
I am trying to use run_turnkey on several subjects (sessions) in parallel. I use version 0.98.1.

First, if I give the sessions parameter a comma-separated list, i.e. sessions=001,002, then the pipeline searches and creates a directory named “001,002” instead of two directories “001” and “002”.

An example call:

qunex_container run_turnkey \
    --rawdatainput="${RAW_DATA}" \
    --paramfile="${INPUT_BATCH_FILE}" \
    --mappingfile="${INPUT_MAPPING_FILE}" \
    --workingdir="${WORK_DIR}" \
    --projectname="${STUDY_NAME}" \
    --path="${WORK_DIR}/${STUDY_NAME}" \
    --sessions="${SESSIONS}" \
    --sessionsfolder="${STUDY_FOLDER}/sessions" \
    --turnkeytype="${RUNTURNKEY_TYPE}" \
    --container="${QUNEX_CONTAINER}" \
    --turnkeysteps="${RUNTURNKEY_STEPS}" \
    --parelements=3 \
    --dockeropt="-v ${BIND_FOLDER}:${BIND_FOLDER}"

Where for example SESSIONS=001,002

Another thing I have tried, is to manually use run_turnkey in parallel by running it on single subjects on different screens, for example one time with sessions=001 and another in a parallel screen with sessions=002.
When I do this, I get an error from rsync:

===> RUNNING run_turnkey step ~~~ map_raw_data

 --> Note: /home/docker/volumes/hcppipelines/asaf/sessions/002 not found. Creating one now.

 --> Running turnkey via local: a6df11001e53


 -- Linking DICOMs into /home/docker/volumes/hcppipelines/asaf/sessions/002/inbox


 -- Running rsync: rsync -azH --exclude "*.xml" --exclude "*.gif" --exclude "*.sh" --exclude "*.txt" --exclude ".*" ${CaseInputFile}/ ${QuNexRawInboxDir}

/opt/qunex/bash/qx_utilities/run_turnkey.sh: line 1846:  6436 Segmentation fault      (core dumped) rsync -azH --exclude "*.xml" --exclude "*.gif" --exclude "*.sh" --exclude "*.txt" --exclude ".*" ${CaseInputFile}/ ${QuNexRawInbo
xDir}

----------------------------------------------------------------------------
  --> Batch file transfer check: pass
  --> Mapping file transfer check: pass
  --> DICOM file count in input folder /input/SCANS: 4312
  --> DICOM file count in output folder /home/docker/volumes/hcppipelines/asaf/sessions/002/inbox: 34
  --> DICOM mapping check: fail

ERROR. Something went wrong.

map_raw_data

ERROR. Something went wrong.
 ===> run_turnkey acceptance testing map_raw_data logs for completion.

 ===> run_turnkey acceptance testing found comlog file for map_raw_data step:
      /home/docker/volumes/hcppipelines/asaf/processing/logs/comlogs/error_map_raw_data_002_2024-08-26_07.39.38.979068.log

 ===> ERROR: run_turnkey acceptance test for map_raw_data step failed.

Is there a way to use run_turnkey more efficiently?

Thank you,
Asaf

2

Hi Asaf,

run_turnkey has several design and implementation flaws, which is why we are deprecating it and replacing it with a new functionality that is coming out with version 1.0.0.. This release is staged for later this year or early next year.

Because of the issues, we seldom use run_turnkey ourselves. Our preferred method of processing right now is the step-by-step one (see QuNex quick start using a Docker container — QuNex documentation). It is much more robust and gives you better control over what is happening. You can also inspect if everything went OK after each step. Note that in step-by-step execution, map_raw_data is replaced with import_dicom (or import_hcp, or import_bids if your data is in a different format).

Best, Jure

3

Alright, thanks Jure :slight_smile: