[RESOLVED] Nu_correct error in hcp_freesurfer pipeline

Thomas · July 26, 2024, 12:23pm

Dear all,

I am currently running the HCP preprocessing pipeline with a singularity container. After creating the data structure I ran:

qunex_container run_turnkey \
	--container="${con}" \
	--bind="${qdir}:${qdir}" \
	--dataformat="BIDS" \
	--bidsname="${bidsdir}" \
	--paramfile="${dir_paramfile}" \
	--mappingfile="${dir_mappingfile}" \
	--workingdir="${qdir}" \
	--projectname="${projectname}" \
	--path="${qdir}/${projectname}" \
	--sessions="sub1_2A" \
	--sessionids="sub1_2A" \
	--sessionsfoldername="sessions" \
	--turnkeytype="local" \
	--overwrite="yes" \
    --turnkeysteps="create_session_info,create_batch,setup_hcp,hcp_pre_freesurfer,hcp_freesurfer,hcp_post_freesurfer,hcp_fmri_volume,hcp_fmri_surface" \
	--scheduler="SLURM,jobname=qunex_turnkey,time=48:00:00,cpus-per-task=4,mem-per-cpu=32000"

However I am repeatedly getting the following error during the hcp_freesurfer pipeline:

(...)
running: /opt/freesurfer/freesurfer/mni/bin/nu_estimate_np_and_em -parzen -log -sharpen 0.15 0.01 -iterations 1000 -stop 0.001 -shrink 4 -auto_mask -nonotify -b_spline 1.0e-7 -distance 50 -quiet -execute -clobber -nokeeptmp -tmpdir ./tmp.mri_nu_correct.mni.2862674/0/ ./tmp.mri_nu_correct.mni.2862674/nu0.mnc ./tmp.mri_nu_correct.mni.2862674/nu1.imp

Assertion failed at line 742 in file templates/CachedArray.cc
nu_estimate_np_and_em: crashed while running volume_stats (termination status=256)
nu_correct: crashed while running nu_estimate_np_and_em (termination status=256)
ERROR: nu_correct
Linux dccn-c084.dccn.nl 4.18.0-553.8.1.el8_10.x86_64 #1 SMP Tue Jul 2 07:26:33 EDT 2024 x86_64 GNU/Linux

recon-all -s EC2129_2A exited with ERRORS at Thu Jul 25 22:52:24 CEST 2024
(...)

I thought this might be an issue of a faulty configuration of the container environment (despite --cleanenv being called by default) so I tried deleting all paths to Freesurfer, FSL etc from my local environment. Also, tried the reverse: binding my local software to the containers’ environment with e.g.:

--bash_post="FREESURFERDIR=/opt/freesurfer/7.3.2/bin/freesurfer".

Would anyone now what is going wrong here? Or where to look for the solution?

I attached files with the container environment and the (anonymized) error log file.

Thank you!

Best regards,

Thomas

error_log.txt (24.9 KB)
qunex_environment.txt (7.7 KB)

demsarjure · July 26, 2024, 2:00pm

Hi,

Unfortunately, we have never seen this and the error logs you sent along are not very informative.

Perhaps you could try with an older version, say 0.99.2d that one has a different operating system.

Also, binding in FreeSurfer 7+ will not work as HCP Pipelines support FreeSurfer 6, so try with that maybe.

Best, Jure

Thomas · July 30, 2024, 10:05am

Dear jure,

Thank you for the quick reply! I tried both options and some more things, but it all does not seem to work.
I thought it might help to re-install qunex from source. However the Clone: git clone https://gitlab.qunex.yale.edu/qunex/qunex.git does not work anymore. Would you now a way to install qunex from source?

Thanks again for the effort!

Best regards,

Thomas

demsarjure · July 30, 2024, 2:03pm

Hi Thomas,

What error do you get when you run

git clone https://gitlab.qunex.yale.edu/qunex/qunex.git

The alternative is to clone with ssh, but for this to work, you need to add your public ssh key to QuNex Gitlab (here: Sign in · GitLab)

git clone git@gitlab.qunex.yale.edu:qunex/qunex.git

Best, Jure

demsarjure · July 30, 2024, 2:20pm

Maybe this can help?

github.com/nipreps/fmriprep

nu_correct: crashed while running nu_estimate_np_and_em (termination status=256)

opened 09:32PM - 23 Mar 22 UTC

closed 08:44AM - 13 May 22 UTC

igorvazz

bug

### What happened? Hello, I'm trying to run fmriprep on a singularity contain…er running on a cluster and I'm getting the following error on the html log generated: ``` Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node result["result"] = node.run(updatehash=updatehash) File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run result = self._run_interface(execute=True) File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface return self._run_command(execute) File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command result = self._interface.run(cwd=outdir) File "/opt/conda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 428, in run runtime = self._run_interface(runtime) File "/opt/conda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 822, in _run_interface self.raise_exception(runtime) File "/opt/conda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 749, in raise_exception raise RuntimeError( RuntimeError: Command: recon-all -autorecon1 -noskullstrip -openmp 8 -subjid sub-0002 -sd /data/derivatives/fmriprep/sourcedata/freesurfer -nomotioncor Standard output: Subject Stamp: freesurfer-Linux-centos6_x86_64-stable-pub-v6.0.1-f53a55a Current Stamp: freesurfer-Linux-centos6_x86_64-stable-pub-v6.0.1-f53a55a INFO: SUBJECTS_DIR is /data/derivatives/fmriprep/sourcedata/freesurfer Actual FREESURFER_HOME /opt/freesurfer -rw-rw-r-- 1 ivaz navigator 21860 Mar 23 15:54 /data/derivatives/fmriprep/sourcedata/freesurfer/sub-0002/scripts/recon-all.log Linux barinel057 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux '/opt/freesurfer/bin/recon-all' -> '/data/derivatives/fmriprep/sourcedata/freesurfer/sub-0002/scripts/recon-all.local-copy' #-------------------------------------------- #@# Talairach Wed Mar 23 16:04:02 WET 2022 /data/derivatives/fmriprep/sourcedata/freesurfer/sub-0002/mri mri_nu_correct.mni --no-rescale --i orig.mgz --o orig_nu.mgz --n 1 --proto-iters 1000 --distance 50 /data/derivatives/fmriprep/sourcedata/freesurfer/sub-0002/mri /opt/freesurfer/bin/mri_nu_correct.mni --no-rescale --i orig.mgz --o orig_nu.mgz --n 1 --proto-iters 1000 --distance 50 nIters 1 $Id: mri_nu_correct.mni,v 1.27 2016/02/26 16:19:49 mreuter Exp $ Linux barinel057 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Wed Mar 23 16:04:03 WET 2022 Program nu_correct, built from: Package MNI N3, version 1.12.0, compiled by nicks@terrier (x86_64-unknown-linux-gnu) on 2015-06-19 at 01:25:34 /bin/bc tmpdir is ./tmp.mri_nu_correct.mni.22790 /data/derivatives/fmriprep/sourcedata/freesurfer/sub-0002/mri mri_convert orig.mgz ./tmp.mri_nu_correct.mni.22790/nu0.mnc -odt float mri_convert.bin orig.mgz ./tmp.mri_nu_correct.mni.22790/nu0.mnc -odt float $Id: mri_convert.c,v 1.226 2016/02/26 16:15:24 mreuter Exp $ reading from orig.mgz... TR=2300.00, TE=0.00, TI=0.00, flip angle=0.00 i_ras = (-1, 0, 0) j_ras = (0, 0, -1) k_ras = (0, 1, 0) changing data type from uchar to float (noscale = 0)... writing to ./tmp.mri_nu_correct.mni.22790/nu0.mnc... -------------------------------------------------------- Iteration 1 Wed Mar 23 16:04:04 WET 2022 nu_correct -clobber ./tmp.mri_nu_correct.mni.22790/nu0.mnc ./tmp.mri_nu_correct.mni.22790/nu1.mnc -tmpdir ./tmp.mri_nu_correct.mni.22790/0/ -iterations 1000 -distance 50 [ivaz@barinel057:/data/derivatives/fmriprep/sourcedata/freesurfer/sub-0002/mri/] [2022-03-23 16:04:04] running: /opt/freesurfer/mni/bin/nu_estimate_np_and_em -parzen -log -sharpen 0.15 0.01 -iterations 1000 -stop 0.001 -shrink 4 -auto_mask -nonotify -b_spline 1.0e-7 -distance 50 -quiet -execute -clobber -nokeeptmp -tmpdir ./tmp.mri_nu_correct.mni.22790/0/ ./tmp.mri_nu_correct.mni.22790/nu0.mnc ./tmp.mri_nu_correct.mni.22790/nu1.imp Assertion failed at line 742 in file templates/CachedArray.cc nu_estimate_np_and_em: crashed while running volume_stats (termination status=256) nu_correct: crashed while running nu_estimate_np_and_em (termination status=256) ERROR: nu_correct Linux barinel057 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux recon-all -s sub-0002 exited with ERRORS at Wed Mar 23 16:04:04 WET 2022 For more details, see the log file /data/derivatives/fmriprep/sourcedata/freesurfer/sub-0002/scripts/recon-all.log To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting Standard error: Return code: 1 ``` ### What command did you use? ```shell #!/bin/bash # #SBATCH -A myproject #SBATCH -J fmriprep #SBATCH --time=48:00:00 #SBATCH -n 1 #SBATCH --cpus-per-task=16 #SBATCH --mem-per-cpu=4G #SBATCH -p cpu1 # Queue names you can submit to # Outputs ---------------------------------- #SBATCH -o /veracruz/projects/c/myproject/igor/myproject/log/%x-%A-%a.out #SBATCH -e /veracruz/projects/c/myproject/igor/myproject/log/%x-%A-%a.err #SBATCH --mail-user=myemail@gmail.com #SBATCH --mail-type=ALL # ------------------------------------------ # Import the Singularity module module load Singularity temp=`mktemp -d -p /veracruz/projects/c/myproject/fmriprep/tmp` export TMPDIR=$temp export SINGULARITY_TMPDIR=$temp export SINGULARITYENV_TMPDIR=$temp export SINGULARITY_CACHEDIR=$temp export SINGULARITYENV_CACHEDIR=$temp export SINGULARITYENV_MPLCONFIGDIR=$TMPDIR/matplotlib mkdir -p ${SINGULARITYENV_MPLCONFIGDIR} HOST_WORK=/veracruz/projects/c/myproject/fmriprep/tmp/work mkdir -p ${HOST_WORK} # Set the study folders ADNI export STUDY=/veracruz/projects/c/myproject/study BIDS_DIR="$STUDY/bids_subset" DERIVS_DIR="$BIDS_DIR/derivatives" LOCAL_FREESURFER_DIR="$DERIVS_DIR/freesurfer" # Prepare some writeable bind-mount points. TEMPLATEFLOW_HOST_HOME=/veracruz/projects/c/myproject/igor/.cache/templateflow FMRIPREP_HOST_CACHE=/veracruz/projects/c/myproject/igor/.cache/fmriprep mkdir -p ${TEMPLATEFLOW_HOST_HOME} mkdir -p ${FMRIPREP_HOST_CACHE} # Prepare derivatives folder mkdir -p ${DERIVS_DIR} mkdir -p ${BIDS_DIR}/derivatives/freesurfer mkdir -p ${BIDS_DIR}/derivatives/fmriprep # Make sure FS_LICENSE is defined in the container. export SINGULARITYENV_FS_LICENSE=/veracruz/projects/c/myproject/fmriprep/license.txt FS_LICENSE=/veracruz/projects/c/myproject/fmriprep/license.txt # Designate a templateflow bind-mount point export SINGULARITYENV_TEMPLATEFLOW_HOME="/templateflow" SINGULARITY_CMD="singularity run --cleanenv \ -B $FS_LICENSE:/license \ -B $BIDS_DIR:/data \ -B ${TEMPLATEFLOW_HOST_HOME}:${SINGULARITYENV_TEMPLATEFLOW_HOME} \ -B ${SINGULARITY_TMPDIR}:/tmp \ -B ${LOCAL_FREESURFER_DIR}:/fsdir \ -B $HOST_WORK:/work /veracruz/projects/c/myproject/fmriprep/fmriprep21.0.1.simg" # Parse the participants.tsv file and extract one subject ID from the line corresponding to this SLURM task. subject=$( sed -n -E "$((${SLURM_ARRAY_TASK_ID} + 1))s/sub-(\S*)\>.*/\1/gp" ${BIDS_DIR}/participants.tsv ) # Compose the command line cmd="${SINGULARITY_CMD} /data /data/derivatives/fmriprep participant \ --fs-license-file /license \ --participant-label $subject \ -w /work/ \ -vv \ --omp-nthreads 8 \ --nthreads 12 \ --mem_mb 30000 \ --fs-subjects-dir /fsdir \ --output-spaces MNI152NLin2009cAsym:res-2 anat fsnative fsaverage5 --use-aroma" # Setup done, run the command echo Running task ${SLURM_ARRAY_TASK_ID} echo Commandline: $cmd eval $cmd exitcode=$? # Output results to a table echo "sub-$subject ${SLURM_ARRAY_TASK_ID} $exitcode" \ >> ${SLURM_JOB_NAME}.${SLURM_ARRAY_JOB_ID}.tsv echo Finished tasks ${SLURM_ARRAY_TASK_ID} with exit code $exitcode exit $exitcode ``` ### What version of fMRIPrep are you running? 21.0.1 ### How are you running fMRIPrep? Singularity ### Is your data BIDS valid? Yes ### Are you reusing any previously computed results? No ### Please copy and paste any relevant log output. _No response_ ### Additional information / screenshots I'm also running the .slurm script from a bash script with the following code: ``` #! /bin/bash sbatch --array=1-$(( $( wc -l /veracruz/projects/c/myproject/study/bids_subset/participants.tsv | cut -f1 -d' ' ) - 1 )) /veracruz/projects/c/myproject/fmriprep/sbatch_igor.slurm ```

liningpan · July 30, 2024, 2:58pm

Do you happen to have the TMPDIR environment variable set somewhere in you bash profile / bashrc? We are starting the container with cleanenv flag but bashrc is still loaded by singularity. The reason I’m asking is that I tracked the assertion failure down to these internal functions.

github.com

BIC-MNI/EBTKS/blob/87a90dfc0b7cd45375c925e948f8ec10bdeba0cb/templates/CachedArray.cc#L742


      
              // Have to specify 'trunc' for some versions of the libraries, 
              // otherwise the file may not be created.
              //
              _s.open(path, ios::in | ios::out | ios::trunc);
          
              // Unlink the file so it will be deleted automatically when
              // closed.
              //
              unlink(path);
          
              assert(_s.is_open());
          
              // Create file at requested size
              _s.seekg(_maxNblocks*_blockSize*sizeof(Type));
              _s.put('\0');
            }
          }
          
          template <class Type>
          void
          CachedArray<Type>::_destroy()

github.com

BIC-MNI/EBTKS/blob/87a90dfc0b7cd45375c925e948f8ec10bdeba0cb/src/FileIO.cc#L325


      
              return;
          
            matClose(file);
          
            if (matFileDict.containsKey(file))
              unlink(matFileDict[file]);
          }
          #endif
          
          int
          get_temp_filename(char *pathname)
          {
          #ifdef HAVE_MKSTEMP
            char *tmp_dir = getenv("TMPDIR");
            if (tmp_dir == NULL) {
          #ifdef P_tmpdir
              tmp_dir = P_tmpdir;
          #else
              tmp_dir = "/tmp";
          #endif
            }

Thomas · July 31, 2024, 4:51pm

Dear Jure and liningpan,

Thank you for the effort!

First about the installation from source:
I have tried both options to install from source but I get the following error without ssh key:

fatal: repository 'https://gitlab.qunex.yale.edu/qunex/qunex.git/' not found

And with the ssh key:

Permanently added 'gitlab.qunex.yale.edu,54.82.210.93' (ECDSA) to the list of known hosts.
remote: 
remote: ========================================================================
remote: 
remote: The project you were looking for could not be found or you don't have permission to view it.
remote: 
remote: ========================================================================
remote: 
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Would you know what is going on there?

Also, about the nu_correct error:

Looking at the fmriprep post, I cannot find a clear environmental variable that has been set incorrectly. Also I do not set the TMPDIR in my bashsrc.
My bashrc looks like:

# Change to homedir
cd -L

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

I even tried to run it from a shell session with only the $HOME and the singularity directories in the environment. That still gave the same error. So if you have any more ideas I would be glad to hear it!

Thank you again!

Best,

Thomas

demsarjure · August 1, 2024, 8:07am

Can you try git clone on your local machine? It seems like the server you are on is somehow blocking your access to our GitLab server. I tested both commands (https and ssh) on a couple of systems and they always worked …

For the second part, you could try this:

qunex_container run_turnkey \
	--container="${con}" \
	--bind="${qdir}:${qdir}" \
	--dataformat="BIDS" \
	--bidsname="${bidsdir}" \
	--paramfile="${dir_paramfile}" \
	--mappingfile="${dir_mappingfile}" \
	--workingdir="${qdir}" \
	--projectname="${projectname}" \
	--path="${qdir}/${projectname}" \
	--sessions="sub1_2A" \
	--sessionids="sub1_2A" \
	--sessionsfoldername="sessions" \
	--turnkeytype="local" \
	--overwrite="yes" \
    --turnkeysteps="create_session_info,create_batch,setup_hcp,hcp_pre_freesurfer,hcp_freesurfer,hcp_post_freesurfer,hcp_fmri_volume,hcp_fmri_surface" \
    --bash_post="unset TMPDIR" \
	--scheduler="SLURM,jobname=qunex_turnkey,time=48:00:00,cpus-per-task=4,mem-per-cpu=32000"

This could work if TMPDIR is indeed the problem. Do you maybe have some things in your .bash_profile or .profile?

Best, Jure

demsarjure · August 20, 2024, 2:01pm

Hi Thomas,

Any updates regarding the issue you were having?

Best, Jure

Thomas · September 10, 2024, 10:24am

Dear Jure,

I am sorry for not replying sooner!
This week I picked up the project again, and I think I found a solution.
Adding container_command += "--writable-tmpfs " to the “qunex_container” script did the trick eventually and the container is now running smoothly.

Thank you for all the advice!

Best regards,

Thomas