[RESOLVED] ERROR: --batchfile flag not found

Hi,

I’m trying to install and test the qunex container using the provide sample HCP data but when I run qunex_container run_turnkey I get an error message saying

ERROR: --batchfile flag set but file not found in default locations: /usr/local/qunex/data/HCPA001_batch.txt

It then directs me to /usr/local/qunex/quickstart/processing/logs/comlog/error_run_turnkey_HCPA001_2021-09-09_14.29.0008858079.log which does not exist.

The batchfile does exist at /usr/local/qunex/data/HCPA001_batch.text and has completely open read permissions so I’m not sure why it says it can’t be found. The exact command I’m running is

sudo ./qunex_container run_turnkey --rawdatainput="${RAW_DATA}" --batchfile="${INPUT_BATCH_FILE}" --mappingfile="${INPUT_MAPPING_FILE}" --workingdir="${WORK_DIR}" --projectname="${STUDY_NAME}" --path="${WORK_DIR}/${STUDY_NAME}" --sessions="${SESSIONS}" --sessionids="${SESSIONS}" --sessionsfoldername="sessions" --turnkeytype="${RUNTURNKEY_TYPE}" --container="${QUNEX_CONTAINER}" --turnkeysteps="${RUNTURNKEY_STEPS}"

from /usr/local/qunex. Any thoughts on what might be going on? Please let me know if there is any additional information I could provide to make this more clear.

Thank you,
Graham

Hi!

Welcome to QuNex forums! I believe the issue arises because the system inside the container does not have access to the folder that is sitting outside of the container. To give the container access to outside folders, you have to bind them. With qunex_container you can do this through the bind parameter. For example, to bind the /usr/local/qunex/data/ folder you would specify:

qunex_container
    <all other parameters>
    --bind="/usr/local/qunex/data/:/usr/local/qunex/data/"

The first part of the parameter value (before the colon) defines the path on your local system, and the second part defines what the path will be mapped to inside the container.

You can also bind several folders by separating them with a comma, for example:

    --bind="/usr/local/qunex/data:/qx_data,/usr/local/qunex/scripts:/qx_scripts"

Now, the local folder /usr/local/qunex/data will be mapped to /qx_data inside the container and local folder /usr/local/qunex/scripts will be mapped to /qx_scripts. Meaning that inside QuNex calls you should be using internal, container paths, so /qx_data and /qx_scripts.

Hopefully this resolves your issue. If it does not let me know. If there are still issues in the execution, please also provide all parameter values (values of the system variables INPUT_BATCH_FILE, INPUT_MAPPING_FILE, WORK_DIR ....

Thank you! Unfortunately I’m still getting the same error. I ran:

sudo ./qunex_container run_turnkey --rawdatainput="${RAW_DATA}" --batchfile="${INPUT_BATCH_FILE}" --mappingfile="${INPUT_MAPPING_FILE}" --workingdir="${WORK_DIR}" --projectname="${STUDY_NAME}" --path="${WORK_DIR}/${STUDY_NAME}" --sessions="${SESSIONS}" --sessionids="${SESSIONS}" --sessionsfoldername="sessions" --turnkeytype="${RUNTURNKEY_TYPE}" --container="${QUNEX_CONTAINER}" --turnkeysteps="${RUNTURNKEY_STEPS}" --bind="/usr/local/qunex/data/:/usr/local/qunex/data/"

Which gave the same error:

ERROR: --batchfile flag set but file not found in default locations: /usr/local/qunex/data/HCPA001_batch.txt

===> ERROR during run_turnkey. Check final QuNex error log output:

/usr/local/qunex/quickstart/processing/logs/comlogs/error_run_turnkey_HCPA001_2021-09-10_11.39.0578455525.log

Here are the valuables of the variables in question:

$RAW_DATA = /usr/local/qunex/data
$INPUT_BATCH_FILE = /usr/local/qunex/data/HCPA001_batch.txt
$INPUT_MAPPING_FILE = /usr/local/qunex/data/HCPA001_mapping.txt
$WORK_DIR = /usr/local/qunex
$STUDY_NAME = quickstart
$SESSIONS = HCPA001
$RUNTURNKEY_TYPE = local
$QUNEX_CONTAINER = gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:0.90.6
$RUNTURNKEY_STEPS = create_study,map_raw_data,import_dicom,create_session_info,setup_hcp,create_batch,hcp_pre_freesurfer,hcp_freesurfer,hcp_post_freesurfer,hcp_fmri_volume,hcp_fmri_surface

Please let me know if anything jumps out at you or if you need any other information and thank you for your assistance!

Ah, my bad. For some reason I assumed that you were using a Singularity container, the bind parameter is a particular thing that Singularity uses, so you should remove it. With Docker the story is a bit different, you have two options here.

1. Use the default mapping

By default when running a Docker container, QuNex will map the current folder you are in as /data inside the container. So if you were to cd /usr/local/qunex/ (into your working directory) and then run the qunex_container command from there, /usr/local/qunex/data/ will be known as /data/data inside the container. The first /data comes from the mapping, the second comes from your /data subfolder. The Meaning that your parameters should use /data instead of /usr/local/qunex/data/. For example, $WORK_DIR=/data $RAW_DATA=/data/data, $INPUT_BATCH_FILE=/data/data/HCPA001_batch.txt.

2. Define a custom mapping

Similarly to the bind parameter, you can specify a custom mapping. Since with docker this specification is different, you should use the dockeropt parameter, for mapping docker uses the -v flag. So:

qunex_container
    <all other parameters>
    --dockeropt="-v /usr/local/qunex/:/usr/local/qunex/"

I believe this should resolve your issues, if not let me know and I will investigate your issue further.

1 Like

If the above solution resolved your issues, please let us know, so we can mark this issue as resolved. Thanks!

This worked, thank you!