[RESOLVED] Run_turnkey fails when it hits hcp_fmri_volume

Description:
I’m trying to process the HCP example data using the qunex docker container but it seems that run_turnkey fails when it gets to the hcp_fmri_volume pipeline. The error is not fatal in that the code continues to run through hcp_fmri_surface but it doesn’t produce anything. Can anyone see what the issue is? I’m following these instructions. Please let me know if there is any additional information I can provide to clarify the issue.

Call:
Command (ran from /usr/local/qunex)

[grw201@bcdl2 qunex]$ sudo ./qunex_container run_turnkey --rawdatainput="${RAW_DATA}" --batchfile="${INPUT_BATCH_FILE}" --mappingfile="${INPUT_MAPPING_FILE}" --workingdir="${WORK_DIR}" --projectname="${STUDY_NAME}" --path="${WORK_DIR}/${STUDY_NAME}" --sessions="${SESSIONS}" --sessionids="${SESSIONS}" --sessionsfoldername="sessions" --turnkeytype="${RUNTURNKEY_TYPE}" --container="${QUNEX_CONTAINER}" --turnkeysteps="${RUNTURNKEY_STEPS}" --bind="/usr/local/qunex/data/:/usr/local/qunex/data/"

Variables:

$RAW_DATA = /usr/local/qunex/data
$INPUT_BATCH_FILE = /usr/local/qunex/data/HCPA001_batch.txt
$INPUT_MAPPING_FILE = /usr/local/qunex/data/HCPA001_mapping.txt
$WORK_DIR = /usr/local/qunex
$STUDY_NAME = quickstart
$SESSIONS = HCPA001
$RUNTURNKEY_TYPE = local
$QUNEX_CONTAINER = gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:0.90.6
$RUNTURNKEY_STEPS = create_study,map_raw_data,import_dicom,create_session_info,setup_hcp,create_batch,hcp_pre_freesurfer,hcp_freesurfer,hcp_post_freesurfer,hcp_fmri_volume,hcp_fmri_surface

Logs:

error_run_turnkey_HCPA001_2021-09-17_17.17.0153489930.log (51.3 KB)

Hi!

Welcome to QuNex forums! It seems like something went wrong inside hcp_fmri_volume. It would be helpful for us to diagnose if you could share the log file for hcp_fmri_volume with us. It should be something like this [tmp/done/error]_hcp_fmri_volume_2_HCPA001_2021-09-17_17.19.1631913585.log in comlog folder. Or you can zip the log folder, if it’s easier for you.

Here you go! There is only a done file hcp_fmir_volume2… not a tmp or error. Let me know if you need anything else!

done_hcp_fmri_volume_2_HCPA001_2021-09-17_17.19.1631913585.log (100.3 KB)

Is there a file for hcp_fmri_volume_1?

Yes, there is a error file for volume 1 but not a done or tmp file.

error_hcp_fmri_volume_1_HCPA001_2021-09-17_17.19.1631913585.log (10.1 KB)

How much memory and storage do you have on the system?

I think about 88GB currently but we have maxed out capacity a few times recently. Do you think that’s it?

Yes. If you take a look at hcp_fmri_volume_1, a process was killed with error code 137, which typically means an out-of-memory error.

1 Like

Ah I see, thank you!

88GB should be plenty if qunex is the only job on the system. 2 cores and 16GB RAM in total works fine for the HCP example on our system.

Right now I have qunex installed at /usr/local/qunex. Our server has a small amount of storage allocated to the root directory (/usr/local, /home, etc) with 12TB of attached storage mounted at /bcdlab. Could I keep qunex installed in the root directory but have the all of the actual data in the mounted storage? In other words, can I keep /usr/local/qunex but move /data to /bcdlab and have all qunex outputs written to /bcdlab?

Certainly, you can do that. You just need to change --bind for singularity or --dockeropt for docker to change what host directories are mapped into the container.