[RESOLVED] Error running `hcp_icafix` with `qunex_container` via Docker

Dear QuNex Team,

Description:

I am running qunex version 0.94.11 inside Docker, and hcp_icafix does not run as expected.

Call:

export QUNEX_CONTAINER="gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:0.94.11"

qunex_container hcp_icafix \
    --batchfile=test_study/processing/batch.txt \
    --sessionsfolder=sessions \
    --parsessions=2 \
    --container="${QUNEX_CONTAINER}"

Logs:

--> QuNex will run the command over 8 sessions. It will utilize:

    Maximum sessions run in parallel for a job: 2.
    Maximum elements run in parallel for a session: 1.
    Up to 2 processes will be utilized for a job.

    Job #1 will run sessions: 04,05,06,07,08,09,10,11

bash: /media/HDD/RESEARCH/test_study/processing/logs/batchlogs/qunex_container_command_2022-10-21_13.05.53.567697.sh: No such file or directory

For a workaround, I ran the qunex command inside the Docker container and it works for now.

Also, I have two related questions:

  1. Unlike run_turnkey command, I found that path for batchfile need to be specified related to host, not the container (e.g., setting /data/test_study/processing/batch.txt does not work). Is this a normal behavior?
  2. Is there any plan to incorporate sICA+FIX, MSMAll, and tICA into the run_turnkey command?

I really appreciate your help!

Best regards,
Minho

Hi,

Thanks for reporting this.

  1. This file (/media/HDD/RESEARCH/test_study/processing/logs/batchlogs/qunex_container_command_2022-10-21_13.05.53.567697.sh) is prepared by the qunex_container script and then executed. It seems like for some reason it does not get created. Are you using the latest version of the qunex_container script? Can you manually check the contents of /media/HDD/RESEARCH/test_study/processing/logs/batchlogs/ please? Thanks!

  2. Yes, qunex_container needs the batchfile on the host as it reads the sessions from it to setup parallel jobs. Currently, the workaround for this is to bind the host folders to the same paths inside the container, this way the path will be the same. We are thinking about adding another parameter, e.g., batchfile_host to the qunex_container script so you can specify both the host and within container paths to the batch file. Or automatically binding the host path to the batch file in the container somehow.

  3. Our TODO list includes a complete rework of run_turnkey making it easier to scale and use. However, I cannot give an eta here as we have to execute a bunch of more pressing tasks before this.

Cheers, Jure

1 Like

Hi Jure,

Thanks again for your help.

  1. The file content seems to be okay:
source /opt/qunex/env/qunex_environment.sh
bash /opt/qunex/bin/qunex.sh hcp_icafix --sessionsfolder="sessions" --parsessions="2" --batchfile="processing/batch.txt" --sessions="04,05,06,07,08,09,10,11" 

Actually, based on your suggestion, I re-installed the qunex_container script and new error came up.

This is the folder structure that I try to run ICA+FIX on:

RESEARCH
β”œβ”€β”€ bids
β”œβ”€β”€ test_study (this is the qunex study folder)
└── RAW

The aforementioned error happens when I run the code when I am at RESEARCH folder.

And now qunex returns a new error:

--> QuNex will run the command over 8 sessions. It will utilize:

    Maximum sessions run in parallel for a job: 2.
    Maximum elements run in parallel for a session: 1.
    Up to 2 processes will be utilized for a job.

    Job #1 will run sessions: 04,05,06,07,08,09,10,11

--> unsetting the following environment variables: PATH MATLABPATH PYTHONPATH QUNEXVer TOOLS QUNEXREPO QUNEXPATH QUNEXLIBRARY QUNEXLIBRARYETC TemplateFolder FSL_FIXDIR FREESURFERDIR FREESURFER_HOME FREESURFER_SCHEDULER FreeSurferSchedulerDIR WORKBENCHDIR DCMNIIDIR DICMNIIDIR MATLABDIR MATLABBINDIR OCTAVEDIR OCTAVEPKGDIR OCTAVEBINDIR RDIR HCPWBDIR AFNIDIR ANTSDIR PYLIBDIR FSLDIR FSLGPUDIR PALMDIR QUNEXMCOMMAND HCPPIPEDIR CARET7DIR GRADUNWARPDIR HCPPIPEDIR_Templates HCPPIPEDIR_Bin HCPPIPEDIR_Config HCPPIPEDIR_PreFS HCPPIPEDIR_FS HCPPIPEDIR_PostFS HCPPIPEDIR_fMRISurf HCPPIPEDIR_fMRIVol HCPPIPEDIR_tfMRI HCPPIPEDIR_dMRI HCPPIPEDIR_dMRITract HCPPIPEDIR_Global HCPPIPEDIR_tfMRIAnalysis HCPCIFTIRWDIR MSMBin HCPPIPEDIR_dMRITractFull HCPPIPEDIR_dMRILegacy AutoPtxFolder FSL_GPU_SCRIPTS FSLGPUBinary EDDYCUDADIR USEOCTAVE QUNEXENV CONDADIR MSMBINDIR MSMCONFIGDIR R_LIBS FSL_FIX_CIFTIRW FSFAST_HOME SUBJECTS_DIR MINC_BIN_DIR MNI_DIR MINC_LIB_DIR MNI_DATAPATH FSF_OUTPUT_FORMAT

Generated by QuNex
------------------------------------------------------------------------
Version: 0.94.11
User: root
System: 804ce9fdc2a0
OS: RedHat Linux #142~18.04.1-Ubuntu SMP Thu Sep 1 16:25:16 UTC 2022
------------------------------------------------------------------------

        β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ\                  β•‘      β–ˆβ–ˆ\   β–ˆβ–ˆ\
       β–ˆβ–ˆ  __β–ˆβ–ˆ\                 β•‘      β–ˆβ–ˆβ–ˆ\  β–ˆβ–ˆ |
       β–ˆβ–ˆ /  β–ˆβ–ˆ |β–ˆβ–ˆ\   β–ˆβ–ˆ\       β•‘      β–ˆβ–ˆβ–ˆβ–ˆ\ β–ˆβ–ˆ | β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ\ β–ˆβ–ˆ\   β–ˆβ–ˆ\
       β–ˆβ–ˆ |  β–ˆβ–ˆ |β–ˆβ–ˆ |  β–ˆβ–ˆ |      β•‘      β–ˆβ–ˆ β–ˆβ–ˆ\β–ˆβ–ˆ |β–ˆβ–ˆ  __β–ˆβ–ˆ\\β–ˆβ–ˆ\ β–ˆβ–ˆ  |
       β–ˆβ–ˆ |  β–ˆβ–ˆ |β–ˆβ–ˆ |  β–ˆβ–ˆ |      β•‘      β–ˆβ–ˆ \β–ˆβ–ˆβ–ˆβ–ˆ |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ |\β–ˆβ–ˆβ–ˆβ–ˆ  /
       β–ˆβ–ˆ β–ˆβ–ˆ\β–ˆβ–ˆ |β–ˆβ–ˆ |  β–ˆβ–ˆ |      β•‘      β–ˆβ–ˆ |\β–ˆβ–ˆβ–ˆ |β–ˆβ–ˆ   ____|β–ˆβ–ˆ  β–ˆβ–ˆ\
       \β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ / \β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ  |      β•‘      β–ˆβ–ˆ | \β–ˆβ–ˆ |\β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ\β–ˆβ–ˆ  /\β–ˆβ–ˆ\
        \___β–ˆβ–ˆβ–ˆ\  \______/       β•‘      \__|  \__| \_______\__/  \__|
            \___|                β•‘


                       DEVELOPED & MAINTAINED BY:

                    Anticevic Lab, Yale University
               Mind & Brain Lab, University of Ljubljana
                     Murray Lab, Yale University

                      COPYRIGHT & LICENSE NOTICE:

Use of this software is subject to the terms and conditions defined in
'LICENSES' which is a part of the QuNex Suite source code package:


--- Full QuNex call for command: hcp_icafix

gmri hcp_icafix --sessionsfolder="sessions" --parsessions="2" --batchfile="test_study/processing/batch.txt" --sessions="04,05,06,07,08,09,10,11"

---------------------------------------------------------



--------------------==== QuNex failed! ====--------------------
ERROR: Execution of qunex command hcp_icafix failed!
ERROR: The specified session file is not found! [test_study/processing/batch.txt]!

The call received was:
(please note that when run through scheduler, all possible parameters,
even non relevant ones are passed)

qunex hcp_icafix \
    --sessionsfolder=sessions \
    --parsessions=2 \
    --batchfile=test_study/processing/batch.txt \
    --sessions=04,05,06,07,08,09,10,11
Traceback (most recent call last):
  File "/opt/qunex/python/qx_utilities/gmri", line 522, in <module>
    main()
  File "/opt/qunex/python/qx_utilities/gmri", line 491, in main
    runCommand(comm, opts)
  File "/opt/qunex/python/qx_utilities/gmri", line 182, in runCommand
    gp.run(command, args)
  File "/opt/qunex/python/qx_utilities/general/process.py", line 711, in run
    sessions, gpref = gc.get_sessions_list(options['sessions'], filter=options['filter'], sessionids=options['sessionids'], verbose=False)
  File "/opt/qunex/python/qx_utilities/general/core.py", line 260, in get_sessions_list
    raise ValueError("ERROR: The specified session file is not found! [%s]!" % listString)
ValueError: ERROR: The specified session file is not found! [test_study/processing/batch.txt]!

If I change path of the batchfile to processing/batch.txt (staying in the same directory RESEARCH), the error is as follow:

Traceback (most recent call last):
  File "/home/mshin/qunex/qunex_container", line 1088, in <module>
    main()
  File "/home/mshin/qunex/qunex_container", line 868, in main
    g_sessionids = get_sessionids(batchfile, sessions, batchfilter)
  File "/home/mshin/qunex/qunex_container", line 433, in get_sessionids
    raise ValueError("ERROR: The specified file is not found! [%s]!" % batchfile)
ValueError: ERROR: The specified file is not found! [processing/batch.txt]!
  1. Thanks. I will keep this in mind. I guess run_turnkey managed this inside the container so setting the path of batch file was different.

  2. Glad to hear that! I look forward to it!

Can you please try rerunning everything with absolute paths, so provide the full path to each file and folder parameter, starting from the system’s root.

Hi,

I was able to successfully run the hcp_icafix with qunex_container command using the hack that mounts the same volume twice, to the same absolute path (with read-only option) and /data folder.

This hack needed to be done since file paths were recorded relative to /data folder in batch.txt.

# -- Set your working directory
export WORK_DIR="/media/HDD/RESEARCH/test_study"

# -- Specify the container
# -- For Docker use the container name and tag:
export QUNEX_CONTAINER="gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:0.94.11"

# -- Location of previously prepared data
export RAW_DATA="${WORK_DIR}/bids"

# -- Batch parameters file
export INPUT_BATCH_FILE="${WORK_DIR}/${STUDY_NAME}/processing/batch.txt"

# -- Mapping file
export INPUT_MAPPING_FILE="${RAW_DATA}/hcp_mapping.txt"

qunex_container hcp_icafix \
    --batchfile="${INPUT_BATCH_FILE}" \
    --sessionsfolder=sessions \
    --parsessions=2 \
    --dockeropt="-v ${WORK_DIR}:${WORK_DIR}:ro -v ${WORK_DIR}:/data" \
    --container="${QUNEX_CONTAINER}"

Glad to hear that! Yes, sometimes the mess that arises because of host and container paths need to be resolved with solutions that feel a bit hacky :confused:. We will try to find a better solution for this.

1 Like

Dear Jure,

I do appreciate your help!

For the next time, when I start again from creating a study,
I will try binding the host folders to the same paths inside the container (such as --dockeropt="-v ${WORK_DIR}:${WORK_DIR}),
as this hack seems resolving most of the headaches.

Best regards,
Minho