[RESOLVED] Qunex_container create_study permission denied

karin.westin · May 9, 2023, 9:40am

Hi!

I’m currently trying to prepare for dwi processing of HCP data. However, when running
qunex_container create_study \
–studyfolder=“${STUDY_FOLDER}” \
–container=“${QUNEX_CONTAINER}” \

the created study folder comes without any permissions. Hence when I try to run

qunex_container import_hcp
–sessionsfolder=“${STUDY_FOLDER}/sessions”
–sessions=“${SESSIONS}”
–masterinbox=“${RAW_DATA}”
–archive=“leave”
–container=“${QUNEX_CONTAINER}”

I get

PermissionError: [Errno 13] Permission denied: ‘/home/karin/qunex/dwi_test/processing/logs/batchlogs/qunex_container_command_2023-05-09_11.27.15.080913.sh’

I’ve tried “sudo chmod” in every thinkable way to change permissions but it doesn’t seem to work and the error remains…

I’m on Ubuntu 20.04 LTS, qunex container version 0.97.3

Thanks in advance!
Karin

demsarjure · May 9, 2023, 12:11pm

Hi Karin,

Welcome to the QuNex forums! This seems like a system issue. First, try giving the container full access to the folder you will create through the bind parameter:

qunex_container create_study \
--studyfolder="${STUDY_FOLDER}" \
--bind="${STUDY_FOLDER}:${STUDY_FOLDER}" \
--container="${QUNEX_CONTAINER}"

Instead of study folder, probably better to bind the root of the study folder, the folder where study is in. Note that you also need to bind folders from which you are importing data and so on. By default containers do not have access to all locations on disks.

Also, when pasting commands, please encapsulate them with triple back tick (`) on the forum so it is easier to read, e.g.:

qunex_container create_study \
--studyfolder="${STUDY_FOLDER}" \
--bind="${STUDY_FOLDER}:${STUDY_FOLDER}" \
--container="${QUNEX_CONTAINER}"

vs

qunex_container create_study
–studyfolder=“${STUDY_FOLDER}”
–bind=“${STUDY_FOLDER}:${STUDY_FOLDER}”
–container=“${QUNEX_CONTAINER}”

Cheers, Jure

karin.westin · May 9, 2023, 1:57pm

Hi and thanks for your reply!

I will better myself and encapsulate commands!

How should binding of additional folder be put? Should that be:

‘’’ ```
qunex_container create_study
–studyfolder=“${STUDY_FOLDER}”
–bind=“${STUDY_FOLDER}:${STUDY_FOLDER}”, “${DATA_FOLDER}:${DATA_FOLDER}”
–container=“${QUNEX_CONTAINER}”


Thank you in advance!
Karin

demsarjure · May 9, 2023, 2:04pm

Hi Karin,

No worries about the formatting, just a suggestion to make all of our lives easier. In terms of command, try:

qunex_container create_study
 --studyfolder="${STUDY_FOLDER}" \
 --bind="${STUDY_FOLDER}:${STUDY_FOLDER}”,“${DATA_FOLDER}:${DATA_FOLDER}" \
 --container="${QUNEX_CONTAINER}"

Also add the --bind thingy to subsequent calls (import_hcp).

Jure

karin.westin · May 9, 2023, 4:56pm

Hi again

I’m afaid I’m still getting the same permission error even though I’m adding bind to both create_study and import_hcp…

Best regards,
Karin

demsarjure · May 9, 2023, 6:13pm

Hi!

Below are some possible ideas.

For full transparency, can you please provide the values of all variables (STUDY_FOLDER, QUNEX_CONTAINER, RAW_DATA, SESSIONS). What kind of data are your trying to import, DICOMS or unprocessed HCP data?
I assume that create_study finishes successfully? The printout says that everything completed as it should and creates the folder structure in STUDY_FOLDER? Can you paste the printout of:

cd $STUDY_FOLDER
tree

What is the printout of:

ls -l $STUDY_FOLDER

So even if you

sudo chmod -R 777 /home/karin/qunex

The thing says that there is an issue with permissions?

Try this:

qunex import_hcp \
  --sessionsfolder="${STUDY_FOLDER}/sessions" \
  --sessions="${SESSIONS}" \
  --inbox="${RAW_DATA}" \
  --action="copy" \
  --archive="leave" \
  --overwrite="yes" \
  --container="${QUNEX_CONTAINER}"

If it does not work, try adding --logfolder="/home/karin" this will write the logs in your home folder.

Jure

karin.westin · May 10, 2023, 8:24am

Fantastic !

“”" ```
sudo chmod -R 777 /home/karin/qunex


worked! Now at least the permission error is gone! Thank you so much!

While I anyway keep disturbing you with my questions I do have another one... 

Every other time I run a qunex_container command I get

docker: invalid reference format.
See ‘docker run --help’.


I haven't changed anything, I just re-run the same command and it pops up.

Best regards and sorry for all of  the questions,
Karin

demsarjure · May 10, 2023, 8:39am

Hi,

I have never seen this error, which operating system are you using?

Could you provide a full command example, with all the parameters (not as variables, e.g. not $STUDY_FOLDER but /home/karin/qunex/qx_study or similar). Does this error crash the processing, or does it work regardless of it?

Jure

karin.westin · May 10, 2023, 9:18am

Hi!
Thank you so much for your rapid replies!

I’m on Ubuntu 20.04

Here are all commands used:

It occurs now and then, even when just re-running the same command. It crashes the process.

$ docker login gitlab.qunex.yale.edu:5002 -u <username>
$ docker pull gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:0.97.3
$ wget --no-check-certificate -r 'https://drive.google.com/uc?export=download&id=1wdWgKvr67yX5J8pVUa6tBGXNAg3fssWs' -O qunex_container

$ PATH=${HOME}/qunex:${PATH}

$ chmod a+x ${HOME}/qunex/qunex_container

$ export STUDY_FOLDER="${HOME}/qunex/dwi_test"

$ export RAW_HCPDATA="${HOME}/qunex/data"
$ export QUNEX_CONTAINER="gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:0.97.3"
$ export SESSIONS="HCP_D0"
$ qunex_container import_hcp  \
--sessionsfolder="${STUDY_FOLDER}/sessions"\
  --sessions="${SESSIONS}" \
--inbox="${RAW_HCPDATA}"\
 --action="copy"\
 --archive="leave"\
 --overwrite="yes"\
 --container=“${QUNEX_CONTAINER}”

--> QuNex will run the command over 1 sessions. It will utilize:

    Maximum sessions run in parallel for a job: 1.
    Maximum elements run in parallel for a session: 1.
    Up to 1 processes will be utilized for a job.

    Job #1 will run sessions: HCP_D0
$ docker: invalid reference format.
See 'docker run --help'.
'''

demsarjure · May 10, 2023, 10:23am

Your command call looks weird. You can see that it includes some weird double quotes “ and “ instead of ". For a computer those are completely different characters. Try:

export RAW_HCPDATA="${HOME}/qunex/data"

export QUNEX_CONTAINER="gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:0.97.3"

export SESSIONS="HCP_D0"

qunex_container import_hcp  \
  --sessionsfolder="${STUDY_FOLDER}/sessions" \
  --sessions="${SESSIONS}" \
  --inbox="${RAW_HCPDATA}" \
  --action="copy" \
  --archive="leave" \
  --overwrite="yes" \
  --container="${QUNEX_CONTAINER}"

Edit: I see that this is actually my bad, somehow those sneaked into my example command. I fixed that now.

Jure

karin.westin · May 10, 2023, 7:23pm

I’ve managed to run qunex_container import hcp without any errors now! Thank you so much for taking the time to answer so quickly!

Best,
Karin