[RESOLVED] ERROR: No sessions specified to process

Which version are you using? 0.99.2d, I tried to reproduce your issue but setting --hcp_bold_dcmethod="NONE" works for me. Can you also try setting it directly/explicitly in the command call and not in the batch file.

Best, Jure

Hi Jure

I’m not sure if it’s just a Friday thing, but I cannot get it to run individually.

qunex_container hcp_fmri_volume \ --batchfile=${INPUT_BATCH_FILE} \ --sessionsfolder=${SESSIONS_FOLDER} \ --sessions=${SESSIONS}\ --parsessions="10" \ --parelements="4" \ --container="${QUNEX_CONTAINER}" \ --overwrite="yes" \ --hcp_bold_dcmethod="NONE"

It cannot find any sessions:
`

Generated by QuNex 0.99.2 on 2024-05-10_13.13.29.343616#

=================================================================
gmri hcp_fmri_volume
–sessionsfolder=β€œ/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/sessions”
–parsessions=β€œ10”
–parelements=β€œ4”
–overwrite=β€œyes”
–hcp_bold_dcmethod=β€œNONE”
–sessions=β€œ/gpfs/home/tqh20wsu/scratch/qunex2/ADNI/adni_parameters.txt”
–sessionids=β€œ012_S_4643” \

ERROR: No sessions specified to process. Please check your batch file, filtering options or sessionids parameter!
`

Also, looking at my previous log which failed the sessions is not using the correct adni_parameters.txt file where I am sure I passed the correct path, but could be my issue.
--sessions="/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/processing/adni_parameters.txt" \

Thank you, Scott

It seems like you are not using the parameters correctly.

The adni_parameters.txt file is used only for the create_batch command. Once you run that the parameters are injected on top of the batch file while all session info is injected at the bottom.

When you run hcp_fmri_volume, if you want to run over all session in the batch file, just omit the sessions parameters. If you want to run over a subset of sessions that are in the batchfile, specify sessions as a comma separated list of session IDs to filter out only a subset of sessions that you specify.

Best, Jure

Hi Jure

It seems that when I run
qunex_container hcp_fmri_volume \ --batchfile=${BATCHFILE} \ --sessionsfolder=${SESSIONS} \ --parsessions="10" \ --parelements="4" \ --container="${CONTAINER}" \ --overwrite=”yes” \ --hcp_bold_dcmethod="NONE"
to be putting the output and logs within my home directory (or the location of where I run the command from), which seems weird. Why would it be doing this?
This behavior seems to mirror when I was trying before, where I said it didn’t create a log, when in fact it was just in the directory I was in when making the call.

It creates a new folder for the subject, but is empty. I have attached the log but it looks fine now and says completed successfully.
Log-hcp_fmri_volume-2024-05-11_11.16.22.982853.log (1.3 KB)

Thank you, Scott

Again, it seems like you are setting wrong values for some parameters

--sessionsfolder=${SESSIONS}

Should be

--sessionsfolder="/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni"

If you want to run processing only on a couple of sessions that are listed in the batchfile use the --sessions paramater, e.g.:

--sessions="012_S_4643,010_S_5000,<and other IDs>"

Best, Jure

Hi Jure

My apologies for not grasping this issue.

I tried just using --sessions="012_S_4643" but got errors for python creating files. Looks like it tried to create a file structure in my home directory again.

[tqh20wsu@c0009 ~]$ qunex_container hcp_fmri_volume      
--batchfile="/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/processing/adni_parameters.txt"      
--sessions="012_S_4643"     
--parsessions="10"      
--parelements="4"      
--container="/gpfs/home/tqh20wsu/scratch/qunex2/qunex_suite-0.99.2d.sif"      
--overwrite=”yes”

--> QuNex will run the command over 1 sessions. It will utilize:

    Maximum sessions run in parallel for a job: 10.
    Maximum elements run in parallel for a session: 4.
    Up to 40 processes will be utilized for a job.

    Job #1 will run sessions: 012_S_4643
[tqh20wsu@c0008 ~]$ --> unsetting the following environment variables: PATH MATLABPATH PYTHONPATH QUNEXVer TOOLS QUNEXREPO QUNEXPATH QUNEXEXTENSIONS QUNEXLIBRARY QUNEXLIBRARYETC TemplateFolder FSL_FIXDIR FREESURFERDIR FREESURFER_HOME FREESURFER_SCHEDULER FreeSurferSchedulerDIR WORKBENCHDIR DCMNIIDIR DICMNIIDIR MATLABDIR MATLABBINDIR OCTAVEDIR OCTAVEPKGDIR OCTAVEBINDIR RDIR HCPWBDIR AFNIDIR PYLIBDIR FSLDIR FSLBINDIR PALMDIR QUNEXMCOMMAND HCPPIPEDIR CARET7DIR GRADUNWARPDIR HCPPIPEDIR_Templates HCPPIPEDIR_Bin HCPPIPEDIR_Config HCPPIPEDIR_PreFS HCPPIPEDIR_FS HCPPIPEDIR_PostFS HCPPIPEDIR_fMRISurf HCPPIPEDIR_fMRIVol HCPPIPEDIR_tfMRI HCPPIPEDIR_dMRI HCPPIPEDIR_dMRITract HCPPIPEDIR_Global HCPPIPEDIR_tfMRIAnalysis HCPCIFTIRWDIR MSMBin HCPPIPEDIR_dMRITractFull HCPPIPEDIR_dMRILegacy AutoPtxFolder EDDYCUDA USEOCTAVE QUNEXENV CONDADIR MSMBINDIR MSMCONFIGDIR R_LIBS FSL_FIX_CIFTIRW FSFAST_HOME SUBJECTS_DIR MINC_BIN_DIR MNI_DIR MINC_LIB_DIR MNI_DATAPATH FSF_OUTPUT_FORMAT ANTSDIR CUDIMOT

Generated by QuNex
------------------------------------------------------------------------
Version: 0.99.2
User: tqh20wsu
System: c0008.hpc.private
OS: RedHat Linux #1 SMP Thu Jan 25 16:17:31 UTC 2024
------------------------------------------------------------------------

        β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ\                  β•‘      β–ˆβ–ˆ\   β–ˆβ–ˆ\
       β–ˆβ–ˆ  __β–ˆβ–ˆ\                 β•‘      β–ˆβ–ˆβ–ˆ\  β–ˆβ–ˆ |
       β–ˆβ–ˆ /  β–ˆβ–ˆ |β–ˆβ–ˆ\   β–ˆβ–ˆ\       β•‘      β–ˆβ–ˆβ–ˆβ–ˆ\ β–ˆβ–ˆ | β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ\ β–ˆβ–ˆ\   β–ˆβ–ˆ\
       β–ˆβ–ˆ |  β–ˆβ–ˆ |β–ˆβ–ˆ |  β–ˆβ–ˆ |      β•‘      β–ˆβ–ˆ β–ˆβ–ˆ\β–ˆβ–ˆ |β–ˆβ–ˆ  __β–ˆβ–ˆ\\β–ˆβ–ˆ\ β–ˆβ–ˆ  |
       β–ˆβ–ˆ |  β–ˆβ–ˆ |β–ˆβ–ˆ |  β–ˆβ–ˆ |      β•‘      β–ˆβ–ˆ \β–ˆβ–ˆβ–ˆβ–ˆ |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ |\β–ˆβ–ˆβ–ˆβ–ˆ  /
       β–ˆβ–ˆ β–ˆβ–ˆ\β–ˆβ–ˆ |β–ˆβ–ˆ |  β–ˆβ–ˆ |      β•‘      β–ˆβ–ˆ |\β–ˆβ–ˆβ–ˆ |β–ˆβ–ˆ   ____|β–ˆβ–ˆ  β–ˆβ–ˆ\
       \β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ / \β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ  |      β•‘      β–ˆβ–ˆ | \β–ˆβ–ˆ |\β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ\β–ˆβ–ˆ  /\β–ˆβ–ˆ\
        \___β–ˆβ–ˆβ–ˆ\  \______/       β•‘      \__|  \__| \_______\__/  \__|
            \___|                β•‘


                       DEVELOPED & MAINTAINED BY:

                    Anticevic Lab, Yale University
               Mind & Brain Lab, University of Ljubljana
                     Murray Lab, Yale University

                      COPYRIGHT & LICENSE NOTICE:

Use of this software is subject to the terms and conditions defined in
'LICENSES' which is a part of the QuNex Suite source code package:
https://gitlab.qunex.yale.edu/qunex/qunex/-/tree/master/LICENSES

 ---> Setting up Octave


 ........................ Running QuNex v0.99.2 ........................


--- Full QuNex call for command: hcp_fmri_volume

gmri hcp_fmri_volume --parsessions="10" --parelements="4" --overwrite="”yes”" --batchfile="/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/processing/adni_parameters.txt" --sessions="012_S_4643"

---------------------------------------------------------


WARNING: Parameter qx_cifti_tail was not specified. Its value was imputed from parameter hcp_cifti_tail and set to '_Atlas'!
WARNING: Parameter qx_nifti_tail was not specified. Its value was imputed from parameter hcp_nifti_tail and set to ''!
WARNING: Parameter cifti_tail was not specified. Its value was imputed from parameter qx_cifti_tail and set to '_Atlas'!
WARNING: Parameter nifti_tail was not specified. Its value was imputed from parameter qx_nifti_tail and set to ''!

--------------------==== QuNex failed! ====--------------------

ERROR: Execution of command `hcp_fmri_volume` failed!
       Please check documentation for the command (`qunex hcp_fmri_volume --h`)!

The call received was:
(please note that when run through scheduler, all possible parameters,
even non relevant ones are passed)

qunex hcp_fmri_volume \
    --parsessions=10 \
    --parelements=4 \
    --overwrite=”yes” \
    --batchfile=/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/processing/adni_parameters.txt \
    --sessions=012_S_4643

--------------------------------------------------------------
Here's the error as caught by python:

Traceback (most recent call last):
  File "/opt/qunex/python/qx_utilities/gmri", line 649, in <module>
    main()
  File "/opt/qunex/python/qx_utilities/gmri", line 614, in main
    runCommand(comm, opts)
  File "/opt/qunex/python/qx_utilities/gmri", line 274, in runCommand
    gp.run(command, args)
  File "/opt/qunex/python/qx_utilities/general/process.py", line 2083, in run
    os.makedirs(cfolder)
  File "<frozen os>", line 215, in makedirs
  File "<frozen os>", line 215, in makedirs
  File "<frozen os>", line 225, in makedirs
PermissionError: [Errno 13] Permission denied: '/gpfs/home/processing'

For completeness I can making the following calls prior to the above code:

module add python/anaconda/2023.07/3.11.4
module load singularity/3.6.0
PATH=${HOME}/scratch/qunex2:${PATH}

I then tried using what you said for sessionsfolder, but a progress folder holding the log folders and files was created in /gpfs/home/tqh20wsu/scratch/qunex2/ which I wasn’t expecting.

qunex_container hcp_fmri_volume  \
    --batchfile="/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/processing/adni_parameters.txt"  \
    --sessionsfolder="/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni" \
    --parsessions="10"  \
    --parelements="4"  \
    --container="/gpfs/home/tqh20wsu/scratch/qunex2/qunex_suite-0.99.2d.sif" \
     --overwrite=”yes”

Log-hcp_fmri_volume-2024-05-12_00.43.55.857504.log (1.3 KB)

This runs and says it completed but /gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/sessions/012_S_4643/hcp/012_S_4643/MNINonLinear/Results/ remains empty

Prior to recieving your reply, I ran the whole turnkey pipeline again, this time including the required hcp_bold_dcmethod to NONE within the batch file. Everything appeared to complete successfully but there is no output in the results folder for rsfMRI
Log-hcp_fmri_surface-2024-05-11_23.32.28.874859.log (1.2 KB)
Log-hcp_fmri_volume-2024-05-11_23.32.26.582537.log (1.3 KB)

Apologies again, if I am missing the blindingly obvious or making mistakes, but I just cannot seem to get over this final part. Your patience and assistance is greatly appreciated.

Thank you, Scott

Hi Scott,

It could be that container does not have access to the folders you want to process in, also there are some weird double quotes around overwrite in your commands. Also, you are trying many things at the same time, so it is a bit confusing. Let us focus on the command by command execution with the qunex_container script. Once that is working, we can focus on other things you are trying out.

You are still not setting your parameters correctly. For example, you are still using the wrong file for the batchfile parameters. Like I already said, the parameters file is used only for create_batch after that the batchfile is created and you use that.

qunex_container hcp_fmri_volume  \
    --batchfile="/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/processing/batch.txt"  \
    --sessionsfolder="/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/sessions" \
    --hcp_bold_dcmethod="NONE" \
    --parsessions="10"  \
    --parelements="4"  \
    --container="/gpfs/home/tqh20wsu/scratch/qunex2/qunex_suite-0.99.2d.sif" \
    --overwrite="yes"

Some additional questions:

  1. I see you are using --parsessions=10 does this mean that you onboarded many sessions? If that is the case for this to work I assume that you reran everything (from import_dicom to hcp_post_freesurfer) for all of the sessions? If you have only 1 sessions then --parsessions=10 has no effect.

  2. Can you upload the --batchfile for me to inspect? Not the paramfile, but the batchfile. paramfile is the one you are trying to stuff into everything, that one only includes parameters, QuNex works with the batchfile, which is a combination of the paramfile and the info of all onboarded sessions. This file is created with the create_batch file that you successfully executed.

Best, Jure

Good morning Jure,

This is the file from /gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/processing/ there is not an explicit batch.txt file.
adni_parameters.txt (2.3 KB)

This is different to the one I use when setting up the turnkey and stored in /gpfs/home/tqh20wsu/scratch/qunex2/ADNI
adni_parameters.txt (1.4 KB)

I am sure I did not create, place or rename the file in /gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/processing/ and the same naming convention is confusing.

I agree the double quotes on --overwrite are strange, I am not sure why it is doing that as I only give one set of quotes, but it could be because of something between Windows and Linux and the copy/paste. I will keep an eye on this for next time.

  1. I have reran everything again using turnkey, but this was still only for the one subject.
    I am only using --parsessions=10 as I got it from the example code. I am currently only trying to process 012_S_4643 as the sole subject at the moment and will progress to multiple subjects once I know I can produce the outputted files I need. So will omit this next time.

  2. Done, see above.

Thank you so much for your continued assistance, especially at a weekend.

Let me know if anything is unclear or you need anything else.

Kind regards, Scott

Hi,

  1. Please rename /gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/processing/adni_parameters.txt to /gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/processing/batch.txt. It seems like you used a weird name when running create_batch which caused a bunch of confusion in our discussions.

  2. When you were mapping the data, you forgot to map the BOLD image:

21  :T1w             :Accelerated Sagittal MPRAGE
31  :T2w             :Sagittal 3D FLAIR
41  :T2w             :Axial T2 STAR
91  :                :Axial rsfMRI (Eyes Open)

You can see that rs-fMRI is not annotated properly. I did not catch that your mapping file left the BOLD data alone. I believe line 91 should be:

91  :bold1:rest      :Axial rsfMRI (Eyes Open)

You can rerun everything with a fixed mapping file, or you can just fix this in the batch file manually for now and keep note of it for when you try and process everything at scale.

After you did this, run:

qunex_container hcp_fmri_volume  \
    --batchfile="/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/processing/batch.txt"  \
    --sessionsfolder="/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/sessions" \
    --hcp_bold_dcmethod="NONE" \
    --container="/gpfs/home/tqh20wsu/scratch/qunex2/qunex_suite-0.99.2d.sif" \
    --overwrite="yes"

And let me know what happens after you do this.

Best, Jure

Hi Jure

I have done as you asked and got the following output.

[tqh20wsu@c0009 ~]$ qunex_container hcp_fmri_volume      --batchfile="/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/processing/batch.txt"      --sessionsfolder="/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/sessions"     --hcp_bold_dcmethod="NONE"     --container="/gpfs/home/tqh20wsu/scratch/qunex2/qunex_suite-0.99.2d.sif"     --overwrite="yes"

--> QuNex will run the command over 1 sessions. It will utilize:

    Maximum sessions run in parallel for a job: 1.
    Maximum elements run in parallel for a session: 1.
    Up to 1 processes will be utilized for a job.

    Job #1 will run sessions: 012_S_4643
[tqh20wsu@c0009 ~]$ --> unsetting the following environment variables: PATH MATLABPATH PYTHONPATH QUNEXVer TOOLS QUNEXREPO QUNEXPATH QUNEXEXTENSIONS QUNEXLIBRARY QUNEXLIBRARYETC TemplateFolder FSL_FIXDIR FREESURFERDIR FREESURFER_HOME FREESURFER_SCHEDULER FreeSurferSchedulerDIR WORKBENCHDIR DCMNIIDIR DICMNIIDIR MATLABDIR MATLABBINDIR OCTAVEDIR OCTAVEPKGDIR OCTAVEBINDIR RDIR HCPWBDIR AFNIDIR PYLIBDIR FSLDIR FSLBINDIR PALMDIR QUNEXMCOMMAND HCPPIPEDIR CARET7DIR GRADUNWARPDIR HCPPIPEDIR_Templates HCPPIPEDIR_Bin HCPPIPEDIR_Config HCPPIPEDIR_PreFS HCPPIPEDIR_FS HCPPIPEDIR_PostFS HCPPIPEDIR_fMRISurf HCPPIPEDIR_fMRIVol HCPPIPEDIR_tfMRI HCPPIPEDIR_dMRI HCPPIPEDIR_dMRITract HCPPIPEDIR_Global HCPPIPEDIR_tfMRIAnalysis HCPCIFTIRWDIR MSMBin HCPPIPEDIR_dMRITractFull HCPPIPEDIR_dMRILegacy AutoPtxFolder EDDYCUDA USEOCTAVE QUNEXENV CONDADIR MSMBINDIR MSMCONFIGDIR R_LIBS FSL_FIX_CIFTIRW FSFAST_HOME SUBJECTS_DIR MINC_BIN_DIR MNI_DIR MINC_LIB_DIR MNI_DATAPATH FSF_OUTPUT_FORMAT ANTSDIR CUDIMOT

Generated by QuNex
------------------------------------------------------------------------
Version: 0.99.2
User: tqh20wsu
System: c0009.hpc.private
OS: RedHat Linux #1 SMP Thu Jan 25 16:17:31 UTC 2024
------------------------------------------------------------------------

        β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ\                  β•‘      β–ˆβ–ˆ\   β–ˆβ–ˆ\
       β–ˆβ–ˆ  __β–ˆβ–ˆ\                 β•‘      β–ˆβ–ˆβ–ˆ\  β–ˆβ–ˆ |
       β–ˆβ–ˆ /  β–ˆβ–ˆ |β–ˆβ–ˆ\   β–ˆβ–ˆ\       β•‘      β–ˆβ–ˆβ–ˆβ–ˆ\ β–ˆβ–ˆ | β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ\ β–ˆβ–ˆ\   β–ˆβ–ˆ\
       β–ˆβ–ˆ |  β–ˆβ–ˆ |β–ˆβ–ˆ |  β–ˆβ–ˆ |      β•‘      β–ˆβ–ˆ β–ˆβ–ˆ\β–ˆβ–ˆ |β–ˆβ–ˆ  __β–ˆβ–ˆ\\β–ˆβ–ˆ\ β–ˆβ–ˆ  |
       β–ˆβ–ˆ |  β–ˆβ–ˆ |β–ˆβ–ˆ |  β–ˆβ–ˆ |      β•‘      β–ˆβ–ˆ \β–ˆβ–ˆβ–ˆβ–ˆ |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ |\β–ˆβ–ˆβ–ˆβ–ˆ  /
       β–ˆβ–ˆ β–ˆβ–ˆ\β–ˆβ–ˆ |β–ˆβ–ˆ |  β–ˆβ–ˆ |      β•‘      β–ˆβ–ˆ |\β–ˆβ–ˆβ–ˆ |β–ˆβ–ˆ   ____|β–ˆβ–ˆ  β–ˆβ–ˆ\
       \β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ / \β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ  |      β•‘      β–ˆβ–ˆ | \β–ˆβ–ˆ |\β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ\β–ˆβ–ˆ  /\β–ˆβ–ˆ\
        \___β–ˆβ–ˆβ–ˆ\  \______/       β•‘      \__|  \__| \_______\__/  \__|
            \___|                β•‘


                       DEVELOPED & MAINTAINED BY:

                    Anticevic Lab, Yale University
               Mind & Brain Lab, University of Ljubljana
                     Murray Lab, Yale University

                      COPYRIGHT & LICENSE NOTICE:

Use of this software is subject to the terms and conditions defined in
'LICENSES' which is a part of the QuNex Suite source code package:
https://gitlab.qunex.yale.edu/qunex/qunex/-/tree/master/LICENSES

 ---> Setting up Octave


 ........................ Running QuNex v0.99.2 ........................


--- Full QuNex call for command: hcp_fmri_volume

gmri hcp_fmri_volume --sessionsfolder="/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/sessions" --hcp_bold_dcmethod="NONE" --overwrite="yes" --batchfile="/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/processing/batch.txt" --sessions="012_S_4643"

---------------------------------------------------------


WARNING: Parameter qx_cifti_tail was not specified. Its value was imputed from parameter hcp_cifti_tail and set to '_Atlas'!
WARNING: Parameter qx_nifti_tail was not specified. Its value was imputed from parameter hcp_nifti_tail and set to ''!
WARNING: Parameter cifti_tail was not specified. Its value was imputed from parameter qx_cifti_tail and set to '_Atlas'!
WARNING: Parameter nifti_tail was not specified. Its value was imputed from parameter qx_nifti_tail and set to ''!
# Generated by QuNex 0.99.2 on 2024-05-12_18.19.23.420776#
=================================================================
gmri hcp_fmri_volume \
  --sessionsfolder="/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/sessions" \
  --hcp_bold_dcmethod="NONE" \
  --overwrite="yes" \
  --sessions="/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/processing/batch.txt" \
  --sessionids="012_S_4643" \
=================================================================

Starting multiprocessing sessions in /gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/processing/batch.txt with a pool of 1 concurrent processes

---- Running local

Starting processing of sessions 012_S_4643 at Sunday, 12. May 2024 18:19:23
=======================================================================================================================================

------------------------------------------------------------
Session id: 012_S_4643
[started on Sunday, 12. May 2024 18:19:23]
Running HCP fMRI Volume pipeline [LegacyStyleData] ...
---> PreFS results present.
---> FS results present.
---> PostFS results present.

---> Preprocessing settings (unwarpdir, refimage, moveref, seimage) for BOLD 1
     ... phase encoding direction not specified
     ... No distortion correction used
     ... ERROR: bold image missing, searched for ['/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/sessions/012_S_4643/hcp/012_S_4643/unprocessed/BOLD_1/012_S_4643_BOLD_1.nii.gz']!
     ... reference image not used
     ... using the HCPpipelines default BOLD mask

Running 1 BOLD images in parallel
---> ERROR: something missing, skipping this BOLD!

HCP fMRIVolume completed on Sunday, 12. May 2024 18:19:23
------------------------------------------------------------


===> Final report for command hcp_fmri_volume
... 012_S_4643 ---> HCP fMRI Volume: bolds 1 not ready
===> Not all tasks completed fully!

I’m assuming I need more settings within the parameters file.

edit to add: Also, there is no bold or bold1 folder at this location /gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni/sessions/012_S_4643/hcp/012_S_4643/unprocessed/

My mappings.txt has the following, I assume I must have mistyped the 91 as 90. What is the difference between bold:rest and bold1:rest`?

#  HCP mapping file
#  ----------------

20  => T1w
21  => T1w
30  => T2w
31  => T2w
40  => T2w
41  => T2w
90  => bold:rest
151 => bold:rest

Thank you, Scott

90  => bold:rest

Should be 91 => bold:rest. You can remove entries for 20, 30 and 40.

Best to omit manual edits and rerun the inital steps, because you need to copy the data in the HCP folder structure as well. Fix the mapping file as described and rerun the starting steps so session_hcp.txt file, batch.txt and HCP folder structure are properly setup. Please be careful when running create batch, paramfile should point to your adni_parameters.txt (the one with parameters only), while target file (batch file) should point to <study_folder>/sessions/batch.txt.

You do not have to rerun FS pipeline, those results will remain intact.

Best, Jure

Hi Jure

That seems to have worked perfectly.

I am not starting to do multiple subjects and came across an issue with setup_hcp when using the --sessions parameter, it would get a short way in and then hang, not even creating a log.

[tqh20wsu@c0008 ~]$ qunex_container setup_hcp      --sourcefolder="${STUDY_FOLDER}"      --container="${QUNEX_CONTAINER}" --sessions="${SESSIONS}"

--> QuNex will run the command over 6 sessions. It will utilize:

    Maximum sessions run in parallel for a job: 1.
    Maximum elements run in parallel for a session: 1.
    Up to 1 processes will be utilized for a job.

    Job #1 will run sessions: 002_S_1155,002_S_4229,002_S_4799,002_S_5178,003_S_6833,012_S_4643
[tqh20wsu@c0008 ~]$ --> unsetting the following environment variables: PATH MATLABPATH PYTHONPATH QUNEXVer TOOLS QUNEXREPO QUNEXPATH QUNEXEXTENSIONS QUNEXLIBRARY QUNEXLIBRARYETC TemplateFolder FSL_FIXDIR FREESURFERDIR FREESURFER_HOME FREESURFER_SCHEDULER FreeSurferSchedulerDIR WORKBENCHDIR DCMNIIDIR DICMNIIDIR MATLABDIR MATLABBINDIR OCTAVEDIR OCTAVEPKGDIR OCTAVEBINDIR RDIR HCPWBDIR AFNIDIR PYLIBDIR FSLDIR FSLBINDIR PALMDIR QUNEXMCOMMAND HCPPIPEDIR CARET7DIR GRADUNWARPDIR HCPPIPEDIR_Templates HCPPIPEDIR_Bin HCPPIPEDIR_Config HCPPIPEDIR_PreFS HCPPIPEDIR_FS HCPPIPEDIR_PostFS HCPPIPEDIR_fMRISurf HCPPIPEDIR_fMRIVol HCPPIPEDIR_tfMRI HCPPIPEDIR_dMRI HCPPIPEDIR_dMRITract HCPPIPEDIR_Global HCPPIPEDIR_tfMRIAnalysis HCPCIFTIRWDIR MSMBin HCPPIPEDIR_dMRITractFull HCPPIPEDIR_dMRILegacy AutoPtxFolder EDDYCUDA USEOCTAVE QUNEXENV CONDADIR MSMBINDIR MSMCONFIGDIR R_LIBS FSL_FIX_CIFTIRW FSFAST_HOME SUBJECTS_DIR MINC_BIN_DIR MNI_DIR MINC_LIB_DIR MNI_DATAPATH FSF_OUTPUT_FORMAT ANTSDIR CUDIMOT

Generated by QuNex
------------------------------------------------------------------------
Version: 0.99.2
User: tqh20wsu
System: c0008.hpc.private
OS: RedHat Linux #1 SMP Thu Jan 25 16:17:31 UTC 2024
------------------------------------------------------------------------

        β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ\                  β•‘      β–ˆβ–ˆ\   β–ˆβ–ˆ\
       β–ˆβ–ˆ  __β–ˆβ–ˆ\                 β•‘      β–ˆβ–ˆβ–ˆ\  β–ˆβ–ˆ |
       β–ˆβ–ˆ /  β–ˆβ–ˆ |β–ˆβ–ˆ\   β–ˆβ–ˆ\       β•‘      β–ˆβ–ˆβ–ˆβ–ˆ\ β–ˆβ–ˆ | β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ\ β–ˆβ–ˆ\   β–ˆβ–ˆ\
       β–ˆβ–ˆ |  β–ˆβ–ˆ |β–ˆβ–ˆ |  β–ˆβ–ˆ |      β•‘      β–ˆβ–ˆ β–ˆβ–ˆ\β–ˆβ–ˆ |β–ˆβ–ˆ  __β–ˆβ–ˆ\\β–ˆβ–ˆ\ β–ˆβ–ˆ  |
       β–ˆβ–ˆ |  β–ˆβ–ˆ |β–ˆβ–ˆ |  β–ˆβ–ˆ |      β•‘      β–ˆβ–ˆ \β–ˆβ–ˆβ–ˆβ–ˆ |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ |\β–ˆβ–ˆβ–ˆβ–ˆ  /
       β–ˆβ–ˆ β–ˆβ–ˆ\β–ˆβ–ˆ |β–ˆβ–ˆ |  β–ˆβ–ˆ |      β•‘      β–ˆβ–ˆ |\β–ˆβ–ˆβ–ˆ |β–ˆβ–ˆ   ____|β–ˆβ–ˆ  β–ˆβ–ˆ\
       \β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ / \β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ  |      β•‘      β–ˆβ–ˆ | \β–ˆβ–ˆ |\β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ\β–ˆβ–ˆ  /\β–ˆβ–ˆ\
        \___β–ˆβ–ˆβ–ˆ\  \______/       β•‘      \__|  \__| \_______\__/  \__|
            \___|                β•‘


                       DEVELOPED & MAINTAINED BY:

                    Anticevic Lab, Yale University
               Mind & Brain Lab, University of Ljubljana
                     Murray Lab, Yale University

                      COPYRIGHT & LICENSE NOTICE:

Use of this software is subject to the terms and conditions defined in
'LICENSES' which is a part of the QuNex Suite source code package:
https://gitlab.qunex.yale.edu/qunex/qunex/-/tree/master/LICENSES

 ---> Setting up Octave


 ........................ Running QuNex v0.99.2 ........................


--- Full QuNex call for command: setup_hcp

gmri setup_hcp --sourcefolder="/gpfs/home/tqh20wsu/scratch/qunex2/quickstart_adni" --sessions="002_S_1155,002_S_4229,002_S_4799,002_S_5178,003_S_6833,012_S_4643"

---------------------------------------------------------


^C

I got around this issue by first doing create_batch and then calling setup_hcp with this call, which solved my issue:

qunex_container setup_hcp \
    --sessionsfolder=${SESSIONS_FOLDER}} \
    --batchfile=${BATCHFILE}} \
    --container="${QUNEX_CONTAINER}"  

With these steps done, I’m now going to attempt the three freesurfer jobs, heres hoping I get no further issues.

Thank you, Scott

Yes, in create_batch you can use --sourcefolder to run over a single session or --sessionsfolder along with --sessions or --batchfile to run across multiple sessions. You can access the help for commands on our official documentation page, for example for setup_hcp: setup_hcp β€” QuNex documentation. You can also access help through inline help.

Let me know if things are now working.

Best, Jure

Hi Jure

Thank you again for your help and support with getting the ADNI data through the HCP pipeline, it has been invaluable and I will be sure to acknowledge you in and written report I create.

One final question if I may…
If I was to pivot and not use my originally intended toolbox for comparison of the hcp outputted data, would you recommend any particular comparison approach that works well with output from QuNex? I am specifically thinking of Computer Vision / Machine Learning approaches that would find features to classify the differences between say a Cognitive Normal subject and a Alzheimer’s Dementia subject.

Kind regards, Scott

Hi Scott,

Glad we got over the β€œfinish line” :slight_smile:.

The output you will most likely use is the preprocessed bold image that is spewed out by hcp_fmri_volume. Additionally you could denoise it with hcp_icafix and maybe also hcp_msmall (hcp_msmall needs data preprocessed with T2w to work I believe). QuNex offers some optional denoising commands that could further clean up the data (nuissance regression, movement regression …).

At this point you have a dense time series (a time series for each voxel, .dtseries.nii). This usually has 90k voxels with a value taken at each time point. In other words a lot of data (probably too many features for what you are trying to do with ML). So the next step is usually dimensionality reduction (parcellation, functional connectivity, GBC). There are also ML methods for (data-driven) dimensionality reduction and some recent research actually facilitates the dense time seres data.

TLDR; there are many paths you could explore and most of them can give novel insight into fMRI data and brain diseases. A lot of work on this subject has already been done (also on the ADNI dataset).

Best, Jure