[RESOLVED] Is there a way to run qunex on a specific GPU in version 0.99.2?

Use Questions
1

Dear Jure:

I was wondering if there is any way I could assign a specific GPU to a given qunex process? Kind of like export CUDA_VISIBLE_DEVICES in the terminal?

Many thanks as always.

Best,
Ed

2

Hi Ed,

Yes, I imagine that is possible but this is heavily system dependent and I am not sure I can just write down the solution here. There are two things that can help you here. In the examples below, I am assuming you are using the Singularity container (.sif) let me know if you are using Docker and I will amend the examples. Also, for some functionality you might need the latest container version (1.0.0).

First, you can enter the container interactively and work within it. Doing so, you can inspect what the GPU situation is once inside the container. Below is some example code for that:

# enter the container, ${CONTAINER} points to the .sif file
singularity run --nvccli -u --cleanenv ${CONTAINER} bash

# note that --nvccli requires NVIDIA container toolkit to be installed on the host system
# https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
# this is a relatively new solution and might not be available on your system, try replacing --nvccli with --nv in that case

# inspect devices
nvidia-smi

# set the device
export CUDA_VISIBLE_DEVICES=<set to desired GPU>

Once you have this information, you can use qunex_container in this way:

qunex_container hcp_diffusion \
    ...
    --cuda \
    --bash_post="export CUDA_VISIBLE_DEVICES=<set to desired GPU>" \
    --container=${CONTAINER}

Here, --cuda will add the --nvccli -u to the call, replace --cuda with --nv as above if needed. --bash_post is the code that gets executed after entering the container, but before running the QuNex command. There is also --bash_pre which gets executed before entering the container.

Best, Jure

3

Hi Jure:

Yes I am using singularity. I will give it a try and may bug you again in case I run into any problem;)

Many thanks again!
Ed