[RESOLVED] BOLDPreprocessing

Dear QuNex Experts:

Apologies for my ignorance if this question is simply too trivial. I was trying to perform BOLDPreprocessing as described in this link with the following command:
qunex create_bold_brain_masks --sessions="processing/batch.txt" --sessionsfolder="sessions" --parsessions="1" --parelements="1" --bolds="rest" --overwrite="no"

I have previously successfully performed HCP pipeline using the following script:

PATH=${HOME}/qunex:${PATH}
export STUDY_NAME="cimt"
export WORK_DIR="${HOME}/qunex"
export QUNEX_CONTAINER="${WORK_DIR}/qunexcontainer/qunex_suite-0.93.6.sif" #"gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:0.93.6"
#export QUNEX_CONTAINER="gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:0.93.6"
export RAW_DATA="${WORK_DIR}/data"
export INPUT_BATCH_FILE="${RAW_DATA}/006_batch.txt"
export INPUT_MAPPING_FILE="${RAW_DATA}/006_mapping.txt"
export SESSIONS="001A,001B,001C,001D,002A,002B,002C,003A,003B,003C,003D,004A,004B,004C,006A,006C"
export RUNTURNKEY_TYPE="local"
export export export RUNTURNKEY_STEPS="create_session_info,setup_hcp,create_batch,hcp_pre_freesurfer,hcp_freesurfer,hcp_post_freesurfer,hcp_fmri_volume,hcp_fmri_surface,hcp_diffusion"

qunex_container run_turnkey   --rawdatainput="${RAW_DATA}"   --batchfile="${INPUT_BATCH_FILE}"   --mappingfile="${INPUT_MAPPING_FILE}"   --workingdir="${WORK_DIR}"   --projectname="${STUDY_NAME}"   --path="${WORK_DIR}/${STUDY_NAME}"   --sessions="${SESSIONS}"   --sessionids="${SESSIONS}"   --sessionsfoldername="sessions"   --turnkeytype="${RUNTURNKEY_TYPE}"   --container="${QUNEX_CONTAINER}"  --turnkeysteps="${RUNTURNKEY_STEPS}" --overwrite="yes"

My mapping file is shown below:

#  HCP mapping file
#  ----------------

61              => T1w
81              => T2w
SpinEchoFieldMap_AP  => SE-FM-AP
SpinEchoFieldMap_PA  => SE-FM-PA
rfMRI_REST_AP        => bold:rest:phenc(AP)
rfMRI_REST_PA        => bold:rest:phenc(PA)
SpinEchoFieldMap_AP  => SE-FM-AP
SpinEchoFieldMap_PA  => SE-FM-PA
dMRI_dir98_AP        => DWI:dir98_AP
dMRI_dir98_PA        => DWI:dir98_PA
dMRI_dir99_AP        => DWI:dir99_AP
dMRI_dir99_PA        => DWI:dir99_PA

For some reason, I got the following error after running qunex create_bold_brain_masks:

Creating masks for bold runs ... 


   Files in 'images/functional will be processed.
   Masks will be saved in 'images/boldmasks..
   The command will create a mask identifying actual coverage of the brain for
   each of the specified BOLD files based on its first frame.

   Please note: when mapping the BOLD data, the following parameter is key: 

   --bolds parameter defines which BOLD files are processed based on their
     specification in batch.txt file. Please see documentation for formatting. 
     If the parameter is not specified the default value is 'all' and all BOLD
     files will be processed.

........................................................

Working on BOLD images in: /home/ehui/qunex/cimt/sessions/006C/images
Resulting masks will be in: /home/ehui/qunex/cimt/sessions/006C/images/segmentation/boldmasks

Based on the settings, rest BOLD files will be processed (see --bolds).
If already present, existing masks will not be overwritten (see --overwrite).

Processing 1 BOLDs in parallel

Working on: bold1
    ... bold data missing, skipping bold
Looked for:/home/ehui/qunex/cimt/sessions/006C/images/functional/bold1.nii.gz

Working on: bold2
    ... bold data missing, skipping bold
Looked for:/home/ehui/qunex/cimt/sessions/006C/images/functional/bold2.nii.gz

Bold mask creation completed on Thursday, 21. July 2022 16:08:04
---------------------------------------------------------


===> Final report for command create_bold_brain_masks
... 001A ---> BOLDS done:  0, missing data:  2, failed:  0, processed:  0, skipped:  0

Below is the folder structure after running the qunex_container with turnkeys:

.
├── 1
│   ├── 1_nonlin_norm.wdir
│   ├── DistortionCorrectionAndEPIToT1wReg_FLIRTBBRAndFreeSurferBBRbased
│   ├── MotionCorrection
│   ├── MotionMatrices
│   └── OneStepResampling
├── 2
│   ├── 2_nonlin_norm.wdir
│   ├── DistortionCorrectionAndEPIToT1wReg_FLIRTBBRAndFreeSurferBBRbased
│   ├── MotionCorrection
│   ├── MotionMatrices
│   └── OneStepResampling
├── Diffusion
│   ├── data
│   ├── eddy
│   ├── QC
│   ├── rawdata
│   ├── reg
│   └── topup
├── MNINonLinear
│   ├── fsaverage
│   ├── fsaverage_LR32k
│   ├── Native
│   ├── Results
│   ├── ROIs
│   ├── StructuralQC
│   └── xfms
├── T1w
│   ├── 001A
│   ├── ACPCAlignment
│   ├── BiasFieldCorrection_sqrtT1wXT2w
│   ├── BrainExtraction_FNIRTbased
│   ├── Diffusion
│   ├── fsaverage_LR32k
│   ├── Native
│   ├── Results
│   └── xfms
├── T2w
│   ├── ACPCAlignment
│   ├── BrainExtraction_FNIRTbased
│   ├── T2wToT1wDistortionCorrectAndReg
│   └── xfms
└── unprocessed
    ├── BOLD_1_AP
    ├── BOLD_2_PA
    ├── Diffusion
    ├── SpinEchoFieldMap1
    ├── SpinEchoFieldMap2
    ├── T1w
    └── T2w

Any help would be greatly appreciated! Thank you very much

Best,
Ed

Hi Ed,

Can you please try running this:

qunex create_bold_brain_masks \
  --sessions="/home/ehui/qunex/cimt/processing/batch.txt" \
  --sessionsfolder="/home/ehui/qunex/cimt/sessions" \
  --parsessions="1" \
  --parelements="1" \
  --bolds="rest" \
  --overwrite="no"

You should always use full paths for parameters. With run_turnkey there is a difference, because run_turnkey does not have full paths yet as the first step usually creates the folder structure. We are planning a major rework of run_turnkey which should resolve these discrepancies.

Let me know how it goes.

Jure

Hi Jure:

Many thanks for your swift response as always. I have just tried and similar errors listed below still appeared:

---------------------------------------------------------
Session id: 006C 
[started on Friday, 22. July 2022 08:13:54]
Creating masks for bold runs ... 


   Files in 'images/functional will be processed.
   Masks will be saved in 'images/boldmasks..
   The command will create a mask identifying actual coverage of the brain for
   each of the specified BOLD files based on its first frame.

   Please note: when mapping the BOLD data, the following parameter is key: 

   --bolds parameter defines which BOLD files are processed based on their
     specification in batch.txt file. Please see documentation for formatting. 
     If the parameter is not specified the default value is 'all' and all BOLD
     files will be processed.

........................................................

Working on BOLD images in: /home/ehui/qunex/cimt/sessions/006C/images
Resulting masks will be in: /home/ehui/qunex/cimt/sessions/006C/images/segmentation/boldmasks

Based on the settings, rest BOLD files will be processed (see --bolds).
If already present, existing masks will not be overwritten (see --overwrite).

Processing 1 BOLDs in parallel

Working on: bold1
    ... bold data missing, skipping bold
Looked for:/home/ehui/qunex/cimt/sessions/006C/images/functional/bold1.nii.gz

Working on: bold2
    ... bold data missing, skipping bold
Looked for:/home/ehui/qunex/cimt/sessions/006C/images/functional/bold2.nii.gz

Bold mask creation completed on Friday, 22. July 2022 08:13:54
---------------------------------------------------------


===> Final report for command create_bold_brain_masks
... 006C ---> BOLDS done:  0, missing data:  2, failed:  0, processed:  0, skipped:  0

Based on the folder structure /home/ehui/qunex/cimt/sessions/001A/hcp/001A (see below):

.
├── 1
│   ├── 1_nonlin_norm.wdir
│   ├── DistortionCorrectionAndEPIToT1wReg_FLIRTBBRAndFreeSurferBBRbased
│   ├── MotionCorrection
│   ├── MotionMatrices
│   └── OneStepResampling
├── 2
│   ├── 2_nonlin_norm.wdir
│   ├── DistortionCorrectionAndEPIToT1wReg_FLIRTBBRAndFreeSurferBBRbased
│   ├── MotionCorrection
│   ├── MotionMatrices
│   └── OneStepResampling
├── Diffusion
│   ├── data
│   ├── eddy
│   ├── QC
│   ├── rawdata
│   ├── reg
│   └── topup
├── MNINonLinear
│   ├── fsaverage
│   ├── fsaverage_LR32k
│   ├── Native
│   ├── Results
│   ├── ROIs
│   ├── StructuralQC
│   └── xfms
├── T1w
│   ├── 001A
│   ├── ACPCAlignment
│   ├── BiasFieldCorrection_sqrtT1wXT2w
│   ├── BrainExtraction_FNIRTbased
│   ├── Diffusion
│   ├── fsaverage_LR32k
│   ├── Native
│   ├── Results
│   └── xfms
├── T2w
│   ├── ACPCAlignment
│   ├── BrainExtraction_FNIRTbased
│   ├── T2wToT1wDistortionCorrectAndReg
│   └── xfms
└── unprocessed
    ├── BOLD_1_AP
    ├── BOLD_2_PA
    ├── Diffusion
    ├── SpinEchoFieldMap1
    ├── SpinEchoFieldMap2
    ├── T1w
    └── T2w

Do you think somehow the qunex didn’t name the folder for the bold:rest data correctly? As you can see above, within the unprocessed folder, they were named correctly BOLD_1_AP, but after running the hcp pipeline, the data were then named as 1 instead of BOLD_1_AP. Perhaps, hence the reason the error said the data couldn’t be located? (please refer to the first thread for the mapping that I used) Also, the folder /home/ehui/qunex/cimt/sessions/006C/images/functional was empty.

The same things happen whether I ran qunex or qunex_container.

Please kindly advise. Thanks so much for your help.

Best,
Ed

Hi Ed,

It seems like we both missed a processing step :slight_smile:. map_hcp_data maps data from HCP folder structure into QuNex folder structure which is required for any QuNex analyses after HCP’s Pipelines.

Try running:

qunex map_hcp_data \
  --sessions="/home/ehui/qunex/cimt/processing/batch.txt" \
  --sessionsfolder="/home/ehui/qunex/cimt/sessions"

Followed by the create_bold_brain_masks command.

Best, Jure

Hi Jure:

Sorry I finally got the time to try your suggestion. Everything works beautifully now!

Thanks so much for your help as always!

Best,
Ed

Great! Glad to help. Feel free to contact us if you encounter any additional issues down the line.