[RESOLVED] Docker - Excessive use of CPU when running scripts that use wb_command

Issues
1

Description:

wb_command uses all available threads when OMP_NUM_THREADS had not been set or is set to '', which is default in qunex.
When running multiple qunex docker containers at the same time, this behavior makes CPU stuck.
Specifically, when running wb_command -cifti-resample in SubcorticalProcessing.sh, which is included in hcp_fmri_surface pipeline, my local workstation got stuck for more than 24 hours.

This behavior can be avoided by setting explicitly export OMP_THREADS=1 or some other numbers.

However, I naively thought that either using --bash_pre argument or --envars to set such environment variable would work, but it did not.

Workaround Call:

By inspecting the raw source code for qunex_container, I was able to successfully set OMP_NUM_THREADS with --dockeropt option:

export QUNEX_CONTAINER="gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:0.94.4"

qunex_container hcp_fmri_surface \
  # after multiple options
  --dockeropt="-v \"$(pwd)\":/data -e OMP_NUM_THREADS=1" \
  --container="${QUNEX_CONTAINER}"

Feature Request:

The workaround call above works only for docker and I have no knowledge about how to deal with this for a singularity container.

Therefore, for robustness, I hope qunex to support a separate argument for openmp thread number, like how --parelements or --parsessions is implemented now.

For example, --ompthreads argument that sets OMP_NUM_THREADS inside the qunex container would be very helpful.

Many thanks,
Minho

2

Hi Minho,

did you maybe try the bash_post parameter? bash_pre executes the bash code before entering the QuNex container, bash_post executes it after entering the QuNex container but before running the QuNex command. This is useful when working on HPC systems, when scheduling you might have to run some code once on the compute node, but before you enter the container (e.g., load some modules, such as CUDA or Singularity). I believe bash_post should properly set OMP_THREADS within the system in the container, which is the system that actually runs the command.

Cheers, Jure

1 Like
3

Hi Jure,

Thanks for quick response!

I missed the parameter bash_post before,
and as you said setting env variable using bash_post worked as in the example below:

export QUNEX_CONTAINER="gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:0.94.4"

qunex_container hcp_fmri_surface \
  # after multiple options
  --bash_post="export OMP_NUM_THREADS=1" \
  --container="${QUNEX_CONTAINER}"

Many thanks,
Minho