Hi, I’m running QuNex run_turnkye using docker container and SLURM. I have two questions about changing the locations of log files.
The command I’m running is:
qunex_container run_turnkey \
--rawdatainput="${QUNEX_TMP_DATA}" \
--dataformat="${data_format}" \
--paramfile="${QUNEX_TMP_DATA}/${qx_session_name}_parameters.txt" \
--mappingfile="${QUNEX_TMP_DATA}/${qx_session_name}_mapping.txt" \
--workingdir="${WORK_DIR}" \
--projectname="${STUDY_NAME}" \
--path="${WORK_DIR}/${STUDY_NAME}" \
--sessions="${qx_session_name}" \
--sessionsfoldername="sessions" \
--log="study|session" \
--hcp_dwi_nogpu="True" \
--hcp_filename='userdefined' \
--turnkeytype="${RUNTURNKEY_TYPE}" \
--dockeropt="-v ${WORK_DIR}:${WORK_DIR}" \
--container="${QUNEX_CONTAINER}" \
--turnkeysteps="${runturnkey_steps_from_file}" \
--scheduler="SLURM,cpus-per-task=8,time=48:00:00,mem-per-cpu=2000,jobname=${STUDY_NAME},--output=${log_location}"
-
The first question is about the use of the log parameter. It seems that the logs are created successfully in /processing/logs but there is no folder //logs or any logs in this session folder.
-
The second question is about the file created as a result of using SLURM that contains the output to the terminal. In my case it is created in my home directory, for example:
/home/theoweiss/qunex_container_2023-12-26_10.18.30.378665.txt
I’ve tried using --output parameter but then I got:
------------------------------
#!/bin/sh
#SBATCH --cpus-per-task=8
#SBATCH --time=48:00:00
#SBATCH --mem-per-cpu=2000
#SBATCH --output=/home/PreProcessing/qunex/tmp_data/log.out
#SBATCH --job-name=yalab
#SBATCH -o /home/theoweiss/qunex_container_2023-12-26_13.53.21.575330.txt
#SBATCH -e /home/theoweiss/qunex_container_2023-12-26_13.53.21.575330.txt
docker run -v /home/PreProcessing/qunex:/home/PreProcessing/qunex -v /home/theoweiss:/home/theoweiss gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:0.98.6a bash /home/theoweiss/qunex_container_command_2023-12-26_13.53.21.575154.sh
So I wonder if the ‘output’ parameter is getting reassigned.
Thank you for your time, Theo
Hi Theo,
The logging system we use right now is a bit archaic and unintuitive. By default the logs are put into the processing/logs
folder. In there they are further divided into comlogs
(more detailed logs with all outputs) and runlogs
(a kind of top-level summary logs, or sometimes just command calls). If you want to change the location of these logs into something else (not processing/logs
) then --logfolder
is your best bet. When I run things myself, I usually create --logfolder
for each command, e.g. --logfolder=${study_folder}/processing/logs/hcp_fmri_volume
for hcp_fmri_volume
. In your case, since you are using run_turnkey
, you could create this for each session
that you are processing and sort the logs this way.
-
Can you point me to the part of the documentation about the --log
parameter (--log="study|session"
). I think this will not work as set in your example, it could be that our documentation is written poorly or misleading. Sorry for this.
-
QuNex tries to put the SLURM log into the /processing/logs/batchlogs
by default. If that does not exist yet (e.g., before create_study
) it puts it into home. You can control this with the --output
parmeters. Try changing your example above like this:
--output="both:${log_location}" \
--scheduler="SLURM,cpus-per-task=8,time=48:00:00,mem-per-cpu=2000,jobname=${STUDY_NAME}"
both
denotes that stderr
and stdout
, so both errors and outputs are put into the same file.
We are planning a major rework of the logging structure in Q1 2024. The rework will change the way logs are organized as well as give users better control over what gets put where.
Kind regards, Jure
Thank you very much for your explanation.
-
The use of --log
parameter is described here:
Logging and log files — QuNex documentation
Is it possible to save the comlogs in multiple folders as suggested with the comma or pipe?
-
The /processing/logs/batchlogs
exists but it is empty in my case. I will try the --output
parameter.
Thanks again,
Theo
I believe the study|session
log
functionality works only for a subset of commands, sadly run_turnkey
is not among them. Like I said we are actively working on improving the logging functionality and remove these inconsistencies.
Jure
No worries, the default folders work for me and The --output
parameter seems very helpful.
Thank you,
Theo