Slurm--Singularity--Use of Qunex Containers

Use Questions
1

Hi Jure！
I have just started using QuNex containers to process my data and due to the amount of data I have to process, I would like to be able to run QuNex on a cluster (hpc) using singularity containers. but I am still not able to run QuNex correctly even after I have tried various ways of running it. i really need your help!
①I have tried directly at the login node, executing the following command:

STUDY_FOLDER="${HOME}/qunex/quickstart"
RAW_DATA="${HOME}/qunex/data"
BASH_POST="export STUDY_FOLDER=${STUDY_FOLDER};export RAW_DATA=${RAW_DATA}"
QUNEX_CONTAINER=${HOME}/containers/qunex_suite-1.1.0.sif
qunex_container run_recipe \
  --recipe_file=“${RAW_DATA}/recipe.yaml” \
  --recipe=“quick_start” \
  --bash_post=“${BASH_POST}” \
  --container="${QUNEX_CONTAINER}” \
  ---bind=“${HOME}:${HOME}” \
  --container="${QUNEX_CONTAINER}” \
  ---scheduler="SLURM,jobname=qunex,time=1-00:00:00,cpus-per-task=1,mem-per-cpu=64G”

However, there is no output result, and the terminal interface is as follows:

(base) [jiayf2024@bme_login2 ~]$ qunex_container run_recipe \
>   --recipe_file="${RAW_DATA}/recipe.yaml" \
>   --recipe="quick_start" \
>   --bash_post="${BASH_POST}" \
>   --container="${QUNEX_CONTAINER}" \
>   --bind="${HOME}:${HOME}" \
>   --container="${QUNEX_CONTAINER}" \
>   --scheduler="SLURM,jobname=qunex,time=1-00:00:00,cpus-per-task=1,mem-per-cpu=64G"
    Qunex will schedule 1 jobs
#!/bin/bash
#SBATCH --time=1-00:00:00
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=64G

②I also tried writing ‘qunex_container’ related commands in a newly created ‘.job’ file, but after running for more than 10 seconds, the job ends very quickly without any error reporting or any processing results.The contents of the ‘.job’ job are as follows：

#!/bin/bash
#SBATCH -J qunex
#SBATCH -p bme_cpu
#SBATCH --cpus-per-task=4
#SBATCH --mail-type=fail
#SBATCH -o  job%j.out
#SBATCH -e  job%j.err
#SBATCH -N 1
hostname
date
module load /public/software/modules/tools/singularity/3.5.2
chmod -R 777 /public_bme2/bme-zhanghan/Jiayifan/qunex
singularity run -e -B /public_bme2/bme-zhanghan/Jiayifan/qunex/data:/data /public_bme2/bme-zhanghan/Jiayifan/qunex/qunex_suite-1.1.0.sif
HOME="/public_bme2/bme-zhanghan/Jiayifan"
STUDY_FOLDER="/public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart"
RAW_DATA="/public_bme2/bme-zhanghan/Jiayifan/qunex/data"
BASH_POST="export STUDY_FOLDER=${STUDY_FOLDER};export RAW_DATA=${RAW_DATA}"
QUNEX_CONTAINER="/public_bme2/bme-zhanghan/Jiayifan/qunex/qunex_suite-1.1.0.sif"
qunex_container run_recipe \
  --recipe_file="${RAW_DATA}/recipe.yaml" \
  --recipe="quick_start" \
  --bash_post="${BASH_POST}" \
  --container="${QUNEX_CONTAINER}" \
  --bind="${HOME}:${HOME}" \
  --container="${QUNEX_CONTAINER}" \
  --scheduler="SLURM,jobname=qunex,time=1-00:00:00,cpus-per-task=1,mem-per-cpu=64G"

③I have also tried executing the command after the login node enters the singularity container, at which point the command can be executed with the following output:

(base) [jiayf2024@bme_login2 ~]$ module load /public/software/modules/tools/singularity/3.5.2
(base) [jiayf2024@bme_login2 qunex]$ singularity run -e -B /public_bme2/bme-zhanghan/Jiayifan/qunex/data:/data /public_bme2/bme-zhanghan/Jiayifan/qunex/qunex_suite-1.1.0.sif
$ source /opt/qunex/env/qunex_environment.sh
---> unsetting the following environment variables: PATH MATLABPATH PYTHONPATH QUNEXVer TOOLS QUNEXREPO QUNEXPATH QUNEXEXTENSIONS QUNEXLIBRARY QUNEXLIBRARYETC TemplateFolder FSL_FIXDIR FREESURFERDIR FREESURFER_HOME FREESURFER_SCHEDULER FreeSurferSchedulerDIR WORKBENCHDIR DCMNIIDIR DICMNIIDIR MATLABDIR MATLABBINDIR OCTAVEDIR OCTAVEPKGDIR OCTAVEBINDIR RDIR HCPWBDIR AFNIDIR PYLIBDIR FSLDIR FSLBINDIR PALMDIR QUNEXMCOMMAND HCPPIPEDIR CARET7DIR GRADUNWARPDIR HCPPIPEDIR_Templates HCPPIPEDIR_Bin HCPPIPEDIR_Config HCPPIPEDIR_PreFS HCPPIPEDIR_FS HCPPIPEDIR_FS_CUSTOM HCPPIPEDIR_PostFS HCPPIPEDIR_fMRISurf HCPPIPEDIR_fMRIVol HCPPIPEDIR_tfMRI HCPPIPEDIR_dMRI HCPPIPEDIR_dMRITract HCPPIPEDIR_Global HCPPIPEDIR_tfMRIAnalysis HCPCIFTIRWDIR MSMBin HCPPIPEDIR_dMRITractFull HCPPIPEDIR_dMRILegacy AutoPtxFolder EDDYCUDA USEOCTAVE QUNEXENV CONDADIR MSMBINDIR MSMCONFIGDIR R_LIBS FSL_FIX_CIFTIRW FSFAST_HOME SUBJECTS_DIR MINC_BIN_DIR MNI_DIR MINC_LIB_DIR MNI_DATAPATH FSF_OUTPUT_FORMAT ANTSDIR CUDIMOT

========================================================================
Generated by QuNex
------------------------------------------------------------------------
Version: 1.1.0 [QIO]
User: jiayf2024
System: bme_login2
OS: Debian Linux #1 SMP Mon Dec 7 11:30:56 UTC 2020
------------------------------------------------------------------------

        ██████\                  ║      ██\   ██\                       
       ██  __██\                 ║      ███\  ██ |                      
       ██ /  ██ |██\   ██\       ║      ████\ ██ | ██████\ ██\   ██\    
       ██ |  ██ |██ |  ██ |      ║      ██ ██\██ |██  __██\\██\ ██  |
       ██ |  ██ |██ |  ██ |      ║      ██ \████ |████████ |\████  /    
       ██ ██\██ |██ |  ██ |      ║      ██ |\███ |██   ____|██  ██\     
       \██████ / \██████  |      ║      ██ | \██ |\███████\██  /\██\    
        \___███\  \______/       ║      \__|  \__| \_______\__/  \__|   
            \___|                ║                                      


                       DEVELOPED & MAINTAINED BY:

               Mind & Brain Lab, University of Ljubljana
                       Cho Lab, Yale University

                      COPYRIGHT & LICENSE NOTICE:

Use of this software is subject to the terms and conditions defined in
QuNex LICENSES which can be found in the LICENSES folder of the QuNex
repository or at https://qunex.yale.edu/qunex-registration
========================================================================

---> Setting up Octave 

(qunex) [bme_login2 qunex]$ STUDY_FOLDER="/public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart"
(qunex) [bme_login2 qunex]$ QUNEX_CONTAINER="/public_bme2/bme-zhanghan/Jiayifan/qunex/qunex_suite-1.1.0.sif"
(qunex) [bme_login2 qunex]$ qunex_container create_study \
    --studyfolder="${STUDY_FOLDER}" \
    --container="${QUNEX_CONTAINER}"
/bin/sh: 1: singularity: not found
(qunex) [bme_login2 qunex]$ qunex create_study     --studyfolder="${STUDY_FOLDER}"     --container="${QUNEX_CONTAINER}"

.......................... Running QuNex v1.1.0 [QIO] ..........................


--- Full QuNex call for command: create_study

qunex create_study --studyfolder="/public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart" --container="/public_bme2/bme-zhanghan/Jiayifan/qunex/qunex_suite-1.1.0.sif"

---------------------------------------------------------


ERROR: Extra argument container is not valid! Please check your command!

ERROR in running create_study at gmri:
Invalid arguments
Provided unknown arguments: [container]!

The call received was: 
(please note that when run through scheduler, all possible parameters, 
even non relevant ones are passed) 

qunex create_study \
    --studyfolder=/public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart \
    --container=/public_bme2/bme-zhanghan/Jiayifan/qunex/qunex_suite-1.1.0.sif 

Please run `qunex create_study --help` to get help for the failed command.

(qunex) [bme_login2 qunex]$ qunex create_study --studyfolder="/public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart"

.......................... Running QuNex v1.1.0 [QIO] ..........................


--- Full QuNex call for command: create_study

qunex create_study --studyfolder="/public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart"

---------------------------------------------------------


started running create_study at 2025-03-09 13:45:50, track progress in /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/logs/comlogs/tmp_create_study_2025-03-09_13.45.50.774791.log
call: gmri create_study studyfolder="/public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart"
-----------------------------------------
Running create_study
===================

Creating study folder structure:
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/analysis
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/analysis/scripts
 ... folder exists: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing
 ... folder exists: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/logs
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/logs/batchlogs
 ... folder exists: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/logs/comlogs
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/logs/runlogs
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/lists
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/scripts
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/scenes
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/scenes/QC
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/scenes/QC/T1w
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/scenes/QC/T2w
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/scenes/QC/myelin
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/scenes/QC/BOLD
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/scenes/QC/DWI
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/info
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/info/demographics
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/info/tasks
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/info/stimuli
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/info/bids
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/info/hcpls
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/inbox
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/inbox/MR
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/inbox/EEG
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/inbox/BIDS
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/inbox/HCPLS
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/inbox/behavior
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/inbox/concs
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/inbox/events
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/archive
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/archive/MR
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/archive/EEG
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/archive/BIDS
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/archive/HCPLS
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/archive/behavior
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/specs
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/QC
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/subjects

Preparing template files:
 ... created parameters_example.txt file
 ... created .qunexstudy file

-----------------------------------------
Finished at 2025-03-09 13:45:50

---> Successful completion of task

The third method, while it can execute some simple commands, I’m not sure it’s running on the login node, and although I subsequently added the ‘-scheduler’ parameter, I feel that the addition of that parameter rather prevents the commands from running for the following reasons:

(qunex) [bme_login2 qunex]$ qunex create_study \
> --studyfolder="${STUDY_FOLDER}" \
> --scheduler="SLURM,jobname=qunex,time=1-00:00:00,cpus-per-task=1,mem-per-cpu=64G"

.......................... Running QuNex v1.1.0 [QIO] ..........................


--- Full QuNex call for command: create_study

qunex create_study --studyfolder="/public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart" --scheduler="SLURM,jobname=qunex,time=1-00:00:00,cpus-per-task=1,mem-per-cpu=64G"

---------------------------------------------------------


---> Running scheduler for command create_study 
 
---> submitting create_study 
 
gmri create_study --studyfolder="/public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart" 
 
Submitting:
------------------------------
#!/bin/bash
#SBATCH --time=1-00:00:00
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=64G
#SBATCH --job-name=qunex
#SBATCH -o /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/logs/SLURM_create_study.2025-03-09_13.50.40.712483.log
#SBATCH -e /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/logs/SLURM_create_study.2025-03-09_13.50.40.712483.log
gmri create_study --studyfolder="/public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart"
---> Submitted jobs 
      NA -> create_study

The command does not complete the full build of ‘quickstart’ and the screenshot of the resulting folder is shown below:

屏幕截图 2025-03-09 135741
屏幕截图 2025-03-09 135741398×107 2.26 KB

So I’m very confused about how to use QuNex correctly on a cluster (hpc), and I’d like to be able to submit the job correctly in order for it to run on the compute node, rather than on the landing node as is suspiciously the case with the third method (which can cause big problems for cluster users).

Your help would be much appreciated! Thank you very much.

Best,
YJia

2

Hi YJia,

Welcome to the QuNex forum!

qunex_container is a python script that you should execute on the login node. It can do two things for you.

First, if you provide the --container parameter, it will execute the desired QuNex command in the container (either through docker run or through singularity exec, depending on the type of the container you use).

Second, if you provide the --scheduler parameter with the SLURM directive, it will also prepare the SLURM header (#SBATCH directives on top of the script) and then schedule the prepared script.

When you run it with both options, it should create qunex_container_command_*.sh in your home folder and then sbatch that. So one thing you can do is investigate your home folder for these files and check if they look OK.

Based on what you pasted, I see a couple of things that could be an issue in the first case/traditional QuNex use case. The first one is that your bind and scheduler parameters have a triple dash: ---bind, ---scheduler. The second is that you are providing the --container parameter twice. The third issue is that in your later examples, you are loading the Singularity module manually, which means that when qunex_container tries to use singularity exec there probably is no Singularity as the module is not loaded. I tried fixing all of this here:

# prep vars
HOME="/public_bme2/bme-zhanghan/Jiayifan"
STUDY_FOLDER="${HOME}/qunex/quickstart"
RAW_DATA="${HOME}/qunex/data"

# bash pre is executed on the compute node before going into the container
BASH_PRE="module load /public/software/modules/tools/singularity/3.5.2"

# bash post is executed on the compute node immediately after entering the container
BASH_POST="export STUDY_FOLDER=${STUDY_FOLDER};export RAW_DATA=${RAW_DATA}"

# path to the sif image
QUNEX_CONTAINER=${HOME}/containers/qunex_suite-1.1.0.sif

# execute
qunex_container run_recipe \
  --recipe_file="${RAW_DATA}/recipe.yaml" \
  --recipe="quick_start" \
  --bash_pre="${BASH_PRE}" \
  --bash_post="${BASH_POST}" \
  --bind="${HOME}:${HOME}" \
  --container="${QUNEX_CONTAINER}" \
  --scheduler="SLURM,jobname=qunex,time=1-00:00:00,cpus-per-task=1,mem-per-cpu=64G"

Like you correctly assumed, you can bypass the qunex_container script and do all the work manually, which you tried in your other examples. I propose we first try the proposed solution above, before we get to “workarounds”. Please let me know how it goes.

Best, Jure

1 Like
3

Hi Jure!

First of all, thank you for your patience in replying！

I submitted the script according to your modification in the terminal and in the .job job, respectively, without success, and the relevant results are as follows:

①Submitted in ‘.job’ job:

#!/bin/bash
#SBATCH -J qunex
#SBATCH -p bme_cpu
#SBATCH --cpus-per-task=4
#SBATCH --mail-type=fail
#SBATCH -o  job%j.out
#SBATCH -e  job%j.err
#SBATCH -N 1

# prep vars
HOME="/public_bme2/bme-zhanghan/Jiayifan"
STUDY_FOLDER="${HOME}/qunex/quickstart"
RAW_DATA="${HOME}/qunex/data"

# bash pre is executed on the compute node before going into the container
BASH_PRE="module load /public/software/modules/tools/singularity/3.5.2"

# bash post is executed on the compute node immediately after entering the container
BASH_POST="export STUDY_FOLDER=${STUDY_FOLDER};export RAW_DATA=${RAW_DATA}"

# path to the sif image
QUNEX_CONTAINER=${HOME}/qunex/qunex_suite-1.1.0.sif

# execute
qunex_container run_recipe \
  --recipe_file="${RAW_DATA}/recipe.yaml" \
  --recipe="quick_start" \
  --bash_pre="${BASH_PRE}" \
  --bash_post="${BASH_POST}" \
  --bind="${HOME}:${HOME}" \
  --container="${QUNEX_CONTAINER}" \
  --scheduler="SLURM,jobname=qunex,time=1-00:00:00,cpus-per-task=1,mem-per-cpu=64G"

The ‘qunex_container_command_*.sh’ it generates is as follows:

source /opt/qunex/env/qunex_environment.sh
export STUDY_FOLDER=/public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart;export RAW_DATA=/public_bme2/bme-zhanghan/Jiayifan/qunex/data
bash /opt/qunex/bin/qunex.sh run_recipe --recipe_file="/public_bme2/bme-zhanghan/Jiayifan/qunex/data/recipe.yaml" --recipe="quick_start"

I also tried to submit ‘qunex_container_command_*.sh’ and it reported the following error:

/opt/gridview/slurm/spool/slurmd/job3017314/slurm_script: line 13: /opt/qunex/env/qunex_environment.sh: No such file or directory
bash: /opt/qunex/bin/qunex.sh: No such file or directory

②Execute the command directly from the terminal

(base) [jiayf2024@bme_login2 qunex]$ HOME="/public_bme2/bme-zhanghan/Jiayifan"
(base) [jiayf2024@bme_login2 qunex]$ STUDY_FOLDER="${HOME}/qunex/quickstart"
(base) [jiayf2024@bme_login2 qunex]$ RAW_DATA="${HOME}/qunex/data"
(base) [jiayf2024@bme_login2 qunex]$ BASH_PRE="module load /public/software/modules/tools/singularity/3.5.2"
(base) [jiayf2024@bme_login2 qunex]$ BASH_POST="export STUDY_FOLDER=${STUDY_FOLDER};export RAW_DATA=${RAW_DATA}"
(base) [jiayf2024@bme_login2 qunex]$ QUNEX_CONTAINER=${HOME}/qunex/qunex_suite-1.1.0.sif
(base) [jiayf2024@bme_login2 qunex]$ qunex_container run_recipe \
>   --recipe_file="${RAW_DATA}/recipe.yaml" \
>   --recipe="quick_start" \
>   --bash_pre="${BASH_PRE}" \
>   --bash_post="${BASH_POST}" \
>   --bind="${HOME}:${HOME}" \
>   --container="${QUNEX_CONTAINER}" \
>   --scheduler="SLURM,jobname=qunex,time=1-00:00:00,cpus-per-task=1,mem-per-cpu=64G"
    Qunex will schedule 1 jobs
#!/bin/bash
#SBATCH --time=1-00:00:00
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=64G

As of now, the only feasible action is to execute the following command at the login node：

(base) [jiayf2024@bme_login2 ~]$ module load /public/software/modules/tools/singularity/3.5.2
(base) [jiayf2024@bme_login2 ~]$ singularity run -e -B /public_bme2/bme-zhanghan/Jiayifan/qunex/data:/data /public_bme2/bme-zhanghan/Jiayifan/qunex/qunex_suite-1.1.0.sif
$ source /opt/qunex/env/qunex_environment.sh
---> unsetting the following environment variables: PATH MATLABPATH PYTHONPATH QUNEXVer TOOLS QUNEXREPO QUNEXPATH QUNEXEXTENSIONS QUNEXLIBRARY QUNEXLIBRARYETC TemplateFolder FSL_FIXDIR FREESURFERDIR FREESURFER_HOME FREESURFER_SCHEDULER FreeSurferSchedulerDIR WORKBENCHDIR DCMNIIDIR DICMNIIDIR MATLABDIR MATLABBINDIR OCTAVEDIR OCTAVEPKGDIR OCTAVEBINDIR RDIR HCPWBDIR AFNIDIR PYLIBDIR FSLDIR FSLBINDIR PALMDIR QUNEXMCOMMAND HCPPIPEDIR CARET7DIR GRADUNWARPDIR HCPPIPEDIR_Templates HCPPIPEDIR_Bin HCPPIPEDIR_Config HCPPIPEDIR_PreFS HCPPIPEDIR_FS HCPPIPEDIR_FS_CUSTOM HCPPIPEDIR_PostFS HCPPIPEDIR_fMRISurf HCPPIPEDIR_fMRIVol HCPPIPEDIR_tfMRI HCPPIPEDIR_dMRI HCPPIPEDIR_dMRITract HCPPIPEDIR_Global HCPPIPEDIR_tfMRIAnalysis HCPCIFTIRWDIR MSMBin HCPPIPEDIR_dMRITractFull HCPPIPEDIR_dMRILegacy AutoPtxFolder EDDYCUDA USEOCTAVE QUNEXENV CONDADIR MSMBINDIR MSMCONFIGDIR R_LIBS FSL_FIX_CIFTIRW FSFAST_HOME SUBJECTS_DIR MINC_BIN_DIR MNI_DIR MINC_LIB_DIR MNI_DATAPATH FSF_OUTPUT_FORMAT ANTSDIR CUDIMOT

========================================================================
Generated by QuNex
------------------------------------------------------------------------
Version: 1.1.0 [QIO]
User: jiayf2024
System: bme_login2
OS: Debian Linux #1 SMP Mon Dec 7 11:30:56 UTC 2020
------------------------------------------------------------------------

        ██████\                  ║      ██\   ██\                       
       ██  __██\                 ║      ███\  ██ |                      
       ██ /  ██ |██\   ██\       ║      ████\ ██ | ██████\ ██\   ██\    
       ██ |  ██ |██ |  ██ |      ║      ██ ██\██ |██  __██\\██\ ██  |
       ██ |  ██ |██ |  ██ |      ║      ██ \████ |████████ |\████  /    
       ██ ██\██ |██ |  ██ |      ║      ██ |\███ |██   ____|██  ██\     
       \██████ / \██████  |      ║      ██ | \██ |\███████\██  /\██\    
        \___███\  \______/       ║      \__|  \__| \_______\__/  \__|   
            \___|                ║                                      


                       DEVELOPED & MAINTAINED BY:

               Mind & Brain Lab, University of Ljubljana
                       Cho Lab, Yale University

                      COPYRIGHT & LICENSE NOTICE:

Use of this software is subject to the terms and conditions defined in
QuNex LICENSES which can be found in the LICENSES folder of the QuNex
repository or at https://qunex.yale.edu/qunex-registration
========================================================================

---> Setting up Octave 

(qunex) [bme_login2 Jiayifan]$ studyfolder="/public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart"
(qunex) [bme_login2 Jiayifan]$ RAW_DATA="/public_bme2/bme-zhanghan/Jiayifan/qunex/data"
(qunex) [bme_login2 Jiayifan]$ SESSIONS="HCPA001"
(qunex) [bme_login2 Jiayifan]$ qunex create_study --studyfolder="/public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart"

.......................... Running QuNex v1.1.0 [QIO] ..........................


--- Full QuNex call for command: create_study

qunex create_study --studyfolder="/public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart"

---------------------------------------------------------


started running create_study at 2025-03-09 20:16:25, track progress in /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/logs/comlogs/tmp_create_study_2025-03-09_20.16.25.020665.log
call: gmri create_study studyfolder="/public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart"
-----------------------------------------
Running create_study
===================

Creating study folder structure:
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/analysis
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/analysis/scripts
 ... folder exists: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing
 ... folder exists: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/logs
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/logs/batchlogs
 ... folder exists: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/logs/comlogs
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/logs/runlogs
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/lists
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/scripts
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/scenes
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/scenes/QC
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/scenes/QC/T1w
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/scenes/QC/T2w
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/scenes/QC/myelin
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/scenes/QC/BOLD
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/processing/scenes/QC/DWI
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/info
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/info/demographics
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/info/tasks
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/info/stimuli
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/info/bids
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/info/hcpls
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/inbox
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/inbox/MR
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/inbox/EEG
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/inbox/BIDS
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/inbox/HCPLS
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/inbox/behavior
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/inbox/concs
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/inbox/events
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/archive
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/archive/MR
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/archive/EEG
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/archive/BIDS
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/archive/HCPLS
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/archive/behavior
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/specs
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/QC
 ... created: /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/subjects

Preparing template files:
 ... created parameters_example.txt file
 ... created .qunexstudy file

-----------------------------------------
Finished at 2025-03-09 20:16:25

---> Successful completion of task

(qunex) [bme_login2 Jiayifan]$ qunex import_dicom     --sessionsfolder="${STUDY_FOLDER}/sessions"     --sessions="${SESSIONS}"     --masterinbox="${RAW_DATA}"
[----------------------------------omit--------------------------------------------------------------------------------]
archived /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/HCPA001/dicom/310
archived /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/HCPA001/dicom/320
archived /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/HCPA001/dicom/330
archived /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/HCPA001/dicom/340
archived /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/HCPA001/dicom/350
archived /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/HCPA001/dicom/360
archived /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/HCPA001/dicom/370
archived /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/HCPA001/dicom/380
archived /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/HCPA001/dicom/390
archived /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/HCPA001/dicom/400
archived /public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart/sessions/HCPA001/dicom/990

Final report
============

Successfully processed:
... HCPA001 [/public_bme2/bme-zhanghan/Jiayifan/qunex/data/HCPA001.zip]

-----------------------------------------
Finished at 2025-03-09 20:09:08

---> Successful completion of task

So, although the last method can be executed correctly, I think he is executing it on the login node, which can cause problems for other users in the cluster (hpc).

I would have preferred a ‘.job’ job submission or a direct job submission in the terminal with the ‘–scheduler’ parameter, but I’m confused considering that neither of these work correctly!

Thank you again for the help!

Best,
YJia

4

I think there was a miss-understanding, if you use qunex_container with the --scheduler parameter, you do not need to prepare any of the SLURM job things (e.g., the SBATCH header above). qunex_container will do that for you … The code I gave you in my previous post should be executed line-by-line directly in the terminal on the login node. Like I said, let us first try going this route and then move to other solutions (like preparing your own .sbatch/job file).

So, you can either put this in a .sh file, make it executable and run it, or you can execute everything line-by-line:

# prep vars
HOME="/public_bme2/bme-zhanghan/Jiayifan"
STUDY_FOLDER="${HOME}/qunex/quickstart"
RAW_DATA="${HOME}/qunex/data"

# bash pre is executed on the compute node before going into the container
BASH_PRE="module load /public/software/modules/tools/singularity/3.5.2"

# bash post is executed on the compute node immediately after entering the container
BASH_POST="export STUDY_FOLDER=${STUDY_FOLDER};export RAW_DATA=${RAW_DATA}"

# path to the sif image
QUNEX_CONTAINER=${HOME}/containers/qunex_suite-1.1.0.sif

# execute
qunex_container run_recipe \
  --recipe_file="${RAW_DATA}/recipe.yaml" \
  --recipe="quick_start" \
  --bash_pre="${BASH_PRE}" \
  --bash_post="${BASH_POST}" \
  --bind="${HOME}:${HOME}" \
  --container="${QUNEX_CONTAINER}" \
  --scheduler="SLURM,jobname=qunex,time=1-00:00:00,cpus-per-task=1,mem-per-cpu=64G"

Best, Jure

1 Like
5

Sorry Jure, I’m really super new to this, so much so that I misinterpreted some of your advice.

I also tried the two suggestions you gave, i.e. line by line in the terminal or in the .sh file

① executed line-by-line directly in the terminal on the login node

(base) [jiayf2024@bme_login2 ~]$ HOME="/public_bme2/bme-zhanghan/Jiayifan"
(base) [jiayf2024@bme_login2 ~]$ STUDY_FOLDER="${HOME}/qunex/quickstart"
(base) [jiayf2024@bme_login2 ~]$ RAW_DATA="${HOME}/qunex/data"
(base) [jiayf2024@bme_login2 ~]$ BASH_PRE="module load /public/software/modules/tools/singularity/3.5.2"
(base) [jiayf2024@bme_login2 ~]$ BASH_POST="export STUDY_FOLDER=${STUDY_FOLDER};export RAW_DATA=${RAW_DATA}"
(base) [jiayf2024@bme_login2 ~]$ QUNEX_CONTAINER=${HOME}/qunex/qunex_suite-1.1.0.sif
(base) [jiayf2024@bme_login2 ~]$ qunex_container run_recipe \
>   --recipe_file="${RAW_DATA}/recipe.yaml" \
>   --recipe="quick_start" \
>   --bash_pre="${BASH_PRE}" \
>   --bash_post="${BASH_POST}" \
>   --bind="${HOME}:${HOME}" \
>   --container="${QUNEX_CONTAINER}" \
>   --scheduler="SLURM,jobname=qunex,time=1-00:00:00,cpus-per-task=1,mem-per-cpu=64G"
    Qunex will schedule 1 jobs
#!/bin/bash
#SBATCH --time=1-00:00:00
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=64G

I’ve tried this before, but the terminal shows that the command failed to execute and doesn’t generate any .out file, only the qunex_container_command_*.sh file, which I’ve also tried to commit, but that didn’t work either. The screenshot of the terminal is as follows：

image
image1021×472 15.6 KB

The content of the qunex_container_command_*.sh file is as follows (its content is exactly the same as before)

source /opt/qunex/env/qunex_environment.sh
export STUDY_FOLDER=/public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart;export RAW_DATA=/public_bme2/bme-zhanghan/Jiayifan/qunex/data
bash /opt/qunex/bin/qunex.sh run_recipe --recipe_file="/public_bme2/bme-zhanghan/Jiayifan/qunex/data/recipe.yaml" --recipe="quick_start"

② I also tried writing what you provided in the .sh script and submitting it, with the same result as in ①.

Thank you again for the help!

Best, YJia

6

Hi,

I found a bug in the qunex_container script that I believe was the reason for the behavior you are seeing. Please update the script with:

# -- Download the script
wget http://jd.mblab.si/qunex/qunex_container

# -- Add to path
PATH=${HOME}/qunex:${PATH}

# -- Make executable
chmod a+x ${HOME}/qunex/qunex_container

As always, please let me know how it goes.

Best, Jure

1 Like
7

Hi Jure，

You’re right！ :sparkling_heart::+1:

I re-downloaded the qunex_container script, and it’s now possible to submit jobs directly from the terminal, and judging by the output logs, it’s running very, very well!

The command I executed in the terminal is as follows:

(base) [jiayf2024@bme_login2 qunex]$ wget http://jd.mblab.si/qunex/qunex_container
--2025-03-10 14:43:54--  http://jd.mblab.si/qunex/qunex_container
Resolving jd.mblab.si (jd.mblab.si)... 193.2.83.76
Connecting to jd.mblab.si (jd.mblab.si)|193.2.83.76|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 45065 (44K) [application/octet-stream]
Saving to: ‘qunex_container’

100%[=============================================================================================================================================================================>] 45,065      76.9KB/s   in 0.6s   

2025-03-10 14:43:56 (76.9 KB/s) - ‘qunex_container’ saved [45065/45065]

(base) [jiayf2024@bme_login2 qunex]$ vim ~/.bashrc
(base) [jiayf2024@bme_login2 qunex]$ vim /public_bme2/bme-zhanghan/Jiayifan/fMRI/.bash_profile
(base) [jiayf2024@bme_login2 qunex]$ source /public_bme2/bme-zhanghan/Jiayifan/fMRI/.bash_profile
(base) [jiayf2024@bme_login2 qunex]$ chmod a+x /public_bme2/bme-zhanghan/Jiayifan/qunex/qunex_container
(base) [jiayf2024@bme_login2 qunex]$ HOME="/public_bme2/bme-zhanghan/Jiayifan"
(base) [jiayf2024@bme_login2 qunex]$ STUDY_FOLDER="${HOME}/qunex/quickstart"
(base) [jiayf2024@bme_login2 qunex]$ RAW_DATA="${HOME}/qunex/data"
(base) [jiayf2024@bme_login2 qunex]$ BASH_PRE="module load /public/software/modules/tools/singularity/3.5.2"
(base) [jiayf2024@bme_login2 qunex]$ BASH_POST="export STUDY_FOLDER=${STUDY_FOLDER};export RAW_DATA=${RAW_DATA}"
(base) [jiayf2024@bme_login2 qunex]$ QUNEX_CONTAINER=${HOME}/qunex/qunex_suite-1.1.0.sif
(base) [jiayf2024@bme_login2 qunex]$ qunex_container run_recipe \
>   --recipe_file="${RAW_DATA}/recipe.yaml" \
>   --recipe="quick_start" \
>   --bash_pre="${BASH_PRE}" \
>   --bash_post="${BASH_POST}" \
>   --bind="${HOME}:${HOME}" \
>   --container="${QUNEX_CONTAINER}" \
>   --scheduler="SLURM,jobname=qunex,time=1-00:00:00,cpus-per-task=1,mem-per-cpu=64G"
    Qunex will schedule 1 jobs

Submitting:
------------------------------
#!/bin/bash
#SBATCH --time=1-00:00:00
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=64G
#SBATCH --job-name=qunex
#SBATCH -o /public_bme2/bme-zhanghan/Jiayifan/qunex_container_2025-03-10_14.46.34.339976.txt
#SBATCH -e /public_bme2/bme-zhanghan/Jiayifan/qunex_container_2025-03-10_14.46.34.339976.txt

module load /public/software/modules/tools/singularity/3.5.2;singularity exec --bind /public_bme2/bme-zhanghan/Jiayifan:/public_bme2/bme-zhanghan/Jiayifan --cleanenv --bind /public_bme2/bme-zhanghan/Jiayifan:/public_bme2/bme-zhanghan/Jiayifan /public_bme2/bme-zhanghan/Jiayifan/qunex/qunex_suite-1.1.0.sif bash /public_bme2/bme-zhanghan/Jiayifan/qunex_container_command_2025-03-10_14.46.34.336501.sh

(base) [jiayf2024@bme_login2 qunex]$ Submitted batch job 3017548
(base) [jiayf2024@bme_login2 qunex]$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           3017548   bme_cpu    qunex jiayf202  R       2:09      1 bme_comput17
(base) [jiayf2024@bme_login2 qunex]$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           3017548   bme_cpu    qunex jiayf202  R       8:14      1 bme_comput17

A screenshot of the terminal is shown below:

image
image1421×865 29.4 KB

The qunex_container_command_*.sh file has the following contents:

source /opt/qunex/env/qunex_environment.sh
export STUDY_FOLDER=/public_bme2/bme-zhanghan/Jiayifan/qunex/quickstart;export RAW_DATA=/public_bme2/bme-zhanghan/Jiayifan/qunex/data
bash /opt/qunex/bin/qunex.sh run_recipe --recipe_file="/public_bme2/bme-zhanghan/Jiayifan/qunex/data/recipe.yaml" --recipe="quick_start"

Once again, my sincere thanks to you! I’ll be exploring the use of QuNex with my data next, so I hope all goes well!

Best, YJia :heart:

8

Yes, things are finally looking good. :slight_smile: Glad it worked!

You are more than welcome to post any questions on this forum if you encounter any issues when you start working with your data.

Best, Jure

1 Like