Hcp_bold_dcmethod parameter setting

Hi, Jure

5/ This is the result of my processing locally, only this one session was processed (even though I set parsessions=4), and the steps before that seem to work fine, except for the hcp_fmri_surface step which ends after a few minutes of running.

# These actually exist.
PATH TO FILE: /home/jiayf/qunex/ADBD/sessions/BDUA_001/hcp/BDUA_001/MNINonLinear
   File was not found

PATH TO FILE: /home/jiayf/qunex/ADBD/sessions/BDUA_001/hcp/BDUA_001/MNINonLinear/fsaverage_LR32k
   File was not found

Best, YJia

Setting parsessions to 4 is not the only requirement. You need to have all the previous steps (including create_session_info, setup_hcp, create_batch) completed for all sessions.

Can you upload your batch file here, so I can check if you have all the sessions in there?

Best, Jure

Hi, Jure

Here’s my batch.txt. Please check it out.

batch.txt (5.4 KB)

Regarding the previously mentioned hcp_fmri_volume, I apologize for misinterpreting the hcp_fmri_volume processing flow. This step does not actually do motion correction, structure-function registration, denoising & filtering and spatial smoothing.

Denoising is usually performed in subsequent spatial and temporal ICAs, and the removal or retention of white matter signals requires me to focus on the sICA+FIX phase of the operation, perhaps changing the sICA classification to include the white matter signals (to avoid removal), and subsequently updating the sICA classification results with ReApplyMRFix.

The hcp_post_fix step will create scene files that can be used to visually review the signal vs. noise classification generated by ICAFIX. So I should change the sICA classification in hcp_icafix?

Best, YJia

Hi,

  1. Your batch file only has a single session (BDUA_001). So that is what wil be processed. Can you paste the create_batch command you used here? Also explore if you have multiple sessions in /home/jiayf/qunex/ADBD/sessions and whether these sessions have a session.txt and session_hcp.txt file in them.

  2. The ICAFix models that are used by default are trained for HCP data. Since your data is not acquired by the HCP acquisition protocol, you will likely need to train your own ICA models in order to get optimal results. See HCP ICAFix pipeline — QuNex documentation for additional details. I assume you can pick which signals/components to remove and which to keep when preparing this model. Note that you will need to use FSL fix for this.

Best, Jure

Hi, Jure

1/ The create_batch command is as follows (I get it from comlogs). In ./ADBD/sessions folder there are indeed multiple sessions, and each of these sessions has its own session.txt and session_hcp.txt file

create_batch sessionsfolder="/home/jiayf/qunex/ADBD/sessions" targetfile="/home/jiayf/qunex/ADBD/processing/batch.txt" overwrite="yes" paramfile="/home/jiayf/qunex/field/HCPA001_parameters.txt" sessions="BDUA_001"

2/ I’m learning what’s in the documentation and it’s very helpful.

Best, YJia

Hi, YJia

Set --overwrite==append to concatenate sessions. Also you need to set `–sessions=<comma_list_of_sessions" and not to a single session.

You also need to run setup_hcp after that as it is setup only for the single sesssion right now.

Best, Jure

Hi, Jure

Do you mean that multiple sessions in the same session folder need to perform the create_batch and setup_hcp operations together?

Considering that there are only two sessionions placed under the session folder for testing, I deleted the other one and kept only BDUA_001. To be on the safe side, I also redid the create_batch and setup_hcp again, but after that the hcp_fmri_surface seems to be in the same condition as before (it ends very quickly and without any obvious error reporting)

Best, YJia

I am addressing this issue:\

This is the result of my processing locally, only this one session was processed (even though I set parsessions=4).

If you have a test study with 1 session, parsessions=4 does not make any sense as there is nothing to parallelieze. Based on above it seems like you expected several sessions to be processed even though you only have 1 session in the study?

Your batch file looks fine for a single session. Based on the above, I thought you have more sessions …

I am not sure what is happening with surface processing as I never saw this issue before and logs do not give us anything to help. The HCP pipelines are triggered fine. It seems like some kind of a weird system issue. Can you try with a different container? A version older or something.

Best, Jure

Hi, Jure

I’m not sure if it’s a problem with one of the intermediate steps, to check I can reprocess this example data separately in SLURM and locally to see if the hcp_fmri_surface step works. If this is still the result, I’ll test it with a lower version of QuNex.

Best, YJia

Hi YJia,

Any updates here, did you manage to get hcp_fmri_surface up and running?

Best, Jure

Hi, Jure

I’ve been a little busy lately and I apologize for not getting back to you in a timely manner.

I’m still using QuNex-1.1.0, and I’m experiencing the same problem when trying to process hcp_fmri_surface both in the cluster and locally. The comlogs show normal operation (done), but the task ends quickly and the contents show some unusual results.

Here are the comlogs for one of the sessions and the batch.txt used for the whole task.

done_hcp_fmri_surface_1_31-JA-JNLA-0081_2025-04-10_10.04.09.327083.log (8.2 KB)
batch.txt (54.6 KB)
batch.txt (54.6 KB)

Regarding PATH TO FILE: /public_bme2/bme-zhanghan/Jiayifan/qunex/SMHC/Group1_1_out/sessions/31-JA-JNLA-0081/hcp/31-JA-JNLA-0081 /MNINonLinear File was not found, I found the folder under that path, the screenshot is as follows

Finally, there is sad news. Regarding import dicom, I tried several settings of --tool, and none of them can complete the format conversion process correctly on UIH data, and currently for UIH data I can only complete the format conversion and overwrite manually. I’m not sure if this is because of the version of dcm2niix?

Best, YJia

Hi.

  1. Do you use --overwrite with hcp_fmri_surface to overwrite any existing results? This seems like a system issue as HCP Pipelines for some reason do not have access to the folder they work in. Do you have --bind=/public_bme2/bme-zhanghan/Jiayifan:/public_bme2/bme-zhanghan/Jiayifan in your command call?

  2. We rely on external tools for .dcm to .nii conversion, if none of them work, then your data is probably not very standard. You said that you can convert this manually, with which tool? A way to onboard this into QuNex is to convert to .nii manually and then organize to BIDS or HCP raw data structure. Next, you can use import_bids or import_hcp commands.

Best, Jure

Hi, Jure

1/ Here is my hcp_fmri_surface command, I added -overwrite=“yes” and -bind=“${HOME}:${HOME}” parameters, but the situation did not improve.

qunex_container hcp_fmri_surface   --sessionsfolder="${STUDY_FOLDER}/sessions"   --batchfile="${STUDY_FOLDER}/processing/batch.txt"   --container="${QUNEX_CONTAINER}"  --bind="${HOME}:${HOME}" --overwrite="yes" --scheduler="SLURM,jobname=fmrisurf,time=1-10:00:00,cpus-per-task=2,mem-per-cpu=50G"

done_hcp_fmri_surface_1_31-JA-JNLA-0032_2025-04-10_15.23.00.242922.log (8.2 KB)

2/ I use dcm2niix in python for format conversion and the format conversion is correct. So I’m wondering why using --tool=“dcm2niix” in import_dicom doesn’t do the conversion correctly.

Like I showed you before, DICOM-Report.txt after import_dicom shows poor results.

# Correct format conversion results
# Generated by QuNex 1.1.0 [QIO] on 2025-04-07_17.29.09.134495
Report for 31-JA-JNLA-0078 (31-JA-JNLA-0078) scanned on 2022-01-27 15:49:31

Scanned at: Shanghai Tongji Hospital
MR device: SIEMENS|Verio|MRC40752

  40     4                        ABI1_t1iso_mprage      1   [TR 2530.00, TE   2.98]   31-JA-JNLA-0078   2022-01-27 15:49:31 
 120    12                           ABI1_bold_rest    240   [TR 2000.00, TE  30.00]   31-JA-JNLA-0078   2022-01-27 16:00:30 
 160    16                          ABI1_diff_hardi     65   [TR 13700.00, TE  85.00]   31-JA-JNLA-0078   2022-01-27 16:27:50 
 220    22                               Mag_Images      1   [TR   28.00, TE  20.00]   31-JA-JNLA-0078   2022-01-27 16:37:35 
 230    23                               Pha_Images      1   [TR   28.00, TE  20.00]   31-JA-JNLA-0078   2022-01-27 16:37:35 
# Incorrect format conversion results
# Generated by QuNex 1.1.0 [QIO] on 2025-03-31_02.21.54.804197
Report for BDUA_001 (PID-20221216-112320-0194) scanned on 2022-12-16 11:42:22

Scanned at: 
MR device: ||

  10     1                     restfMRI_B0map_Echo2     63   [TR  700.00, TE   7.38]   PID-20221216-112320-0194   2022-12-16 11:42:22 
     WARNING: number of frames in nii does not match dicom information: 1 vs. 63 frames
     WARNING: no slice number information, use qunex reslice manually to correct /home/jiayf/qunex/ADBD/sessions/BDUA_001/nii/11.nii.gz
2010   201               t1_gre_fsp_3d_sag_iso1_NIF    192   [TR    6.94, TE   3.00]   PID-20221216-112320-0194   2022-12-16 11:31:09 
     WARNING: number of frames in nii does not match dicom information: 1 vs. 192 frames
     WARNING: no slice number information, use qunex reslice manually to correct /home/jiayf/qunex/ADBD/sessions/BDUA_001/nii/2011.nii.gz
2020   202                   t1_gre_fsp_3d_sag_iso1    192   [TR    6.94, TE   3.00]   PID-20221216-112320-0194   2022-12-16 11:31:17 
     WARNING: number of frames in nii does not match dicom information: 1 vs. 192 frames
     WARNING: no slice number information, use qunex reslice manually to correct /home/jiayf/qunex/ADBD/sessions/BDUA_001/nii/2021.nii.gz
2040   204            t1_gre_fsp_3d_sag_iso1_MPRTRA     51   [TR    6.94, TE   3.00]   PID-20221216-112320-0194   2022-12-16 11:31:18 
     WARNING: number of frames in nii does not match dicom information: 1 vs. 51 frames
     WARNING: no slice number information, use qunex reslice manually to correct /home/jiayf/qunex/ADBD/sessions/BDUA_001/nii/2041.nii.gz
3010   301                      t2_fse_tra_ACS_fast     25   [TR 6136.00, TE 120.00]   PID-20221216-112320-0194   2022-12-16 11:31:10 
     WARNING: number of frames in nii does not match dicom information: 1 vs. 25 frames
     WARNING: no slice number information, use qunex reslice manually to correct /home/jiayf/qunex/ADBD/sessions/BDUA_001/nii/3011.nii.gz
5010   501      restfMRI_mb7_240_iso2.5mm_SaveBySlc    240   [TR 2000.00, TE  30.00]   PID-20221216-112320-0194   2022-12-16 11:40:11 
     WARNING: number of frames in nii does not match dicom information: 48 vs. 240 frames
     WARNING: no slice number information, use qunex reslice manually to correct /home/jiayf/qunex/ADBD/sessions/BDUA_001/nii/5011.nii.gz

Best, YJia

  1. Actually, processing worked fine :slight_smile: . I just noticed that this is a legacy style dataset and the log is perfectly fine. The errors you are seeing are false positives as there are missing T2w files when HCP Pipelines try to prepare a QC scene. You can ignore there, attached is my example with a similar dataset.
NAME OF FILE: T2w_restore.nii.gz
PATH TO FILE: /data/jdemsar/studies/harvard/sessions/002_01/hcp/002_01/MNINonLinear
   File was not found

done_hcp_fmri_surface_BOLD_1_002_01_2025-04-09_06.59.43.996627.log (7.4 KB)

  1. Which version of dcm2niix are you using? It could be that it got updated lately and they upgraded some things that help you out. I can update in the QuNex container for the 1.1.2 release and see if that helps out. Furthermore, there are some hacks that would bind your external dcm2niix over the one in the QuNex container and force QuNex to use the latest one.

Best, Jure

Hi, Jure

1/ Thank you very much, your processing using the legacy style dataset does work the same way as mine, which clears up my confusion.

2/ The lowest version of dcm2niix I’m currently using is dcm2niiX v1.0.20190902 from MRICron, which has no problems with the format conversion results, and all the dicom files are correctly converted to NIfTI format. So I think at least dcm2niix 2019 and its later versions are working. Some people in our group have also encountered this type of error, and they believe that it is usually an error brought about by a lower version of dcm2niix.

Best, YJia

This is weird, QuNex uses

    dcm2niix Binary  : /opt/dcm2niix/dcm2niix/bin/dcm2niix
    dcm2niix Version : Chris Rorden's dcm2niiX version v1.0.20240202  (JP2:OpenJPEG) (JP-LS:CharLS) GCC12.2.0 x86-64 (64-bit Linux)

Can you provide the dcm2niix call that worked for you, so I can compare with the QuNex call and see what can be done.

Did you check the conversion results manually to confirm that your manual conversion does not have the same issues as the one through QuNex (WARNING: number of frames in nii does not match dicom information:).

Hi, Jure

To show you better, I re-found an example of UIH data and used ‘–tool=“dcm2niix”’ to get the import_dicom result. I then processed it using Chris Rorden's dcm2niiX version v1.0.20240202 in python and output the information about the nii file.

The call for the import_dicom command I use is as follows

call: gmri import_dicom sessionsfolder="/home/jiayf/qunex/test/sessions" masterinbox="/home/jiayf/qunex/field" overwrite="yes" tool="dcm2niix" sessions="BDUA_001"

The sample python code used is as follows.

import os
import subprocess
import nibabel as nib
# Chris Rorden's dcm2niiX version v1.0.20240202

def convert_dicom_to_nifti(input_root_dir, output_dir, additional_nii_dir=None):
    """
    Convert DICOM folders in the input root directory to NIfTI format and output essential information and frame counts.
    Also optionally read and output information from additional NIfTI files in a specified directory.

    Parameters:
        input_root_dir (str): Root directory containing multiple DICOM folders.
        output_dir (str): Directory where NIfTI files will be output.
        additional_nii_dir (str, optional): Directory containing additional NIfTI files to read and output information from.
    """
    # Ensure output directory exists
    os.makedirs(output_dir, exist_ok=True)

    # Folder hierarchy: input_root_dir contains multiple DICOM folders
    print(f"\nProcessing DICOM folders from: {input_root_dir}")
    print(f"Output NIfTI files will be saved to: {output_dir}")

    # Iterate through all DICOM folders in the input root directory
    for folder_name in os.listdir(input_root_dir):
        folder_path = os.path.join(input_root_dir, folder_name)
        if os.path.isdir(folder_path):
            # Check if the folder contains DICOM files
            if any(file.endswith('.dcm') for file in os.listdir(folder_path)):
                # Output file name matches the DICOM folder name
                output_nifti_name = f"{folder_name}"
                output_nifti_path = os.path.join(output_dir, f"{folder_name}.nii.gz")

                # Call dcm2niix for conversion
                command = [
                    "dcm2niix",
                    "-z", "y",  # Compressed output
                    "-o", output_dir,  # Output directory
                    "-f", output_nifti_name,  # Output file name
                    folder_path  # Input DICOM folder
                ]
                subprocess.run(command, check=True)

                # Extract and output essential NIfTI file information
                try:
                    nifti_image = nib.load(output_nifti_path)
                    print(f"\n===== {folder_name} =====")
                    print(f"Output file: {output_nifti_path}")
                    print(f"Dimensions: {nifti_image.shape}")
                    print(f"Data type: {nifti_image.get_data_dtype()}")
                    print(f"Voxel size: {nifti_image.header.get_zooms()}")
                    
                    # Extract frame information
                    if len(nifti_image.shape) >= 4:
                        frames = nifti_image.shape[3]
                        print(f"Number of frames: {frames}")
                    else:
                        print("This file does not contain frame information")
                except Exception as e:
                    print(f"Failed to load NIfTI file: {e}")

    # If additional NIfTI directory is specified, process those files as well
    if additional_nii_dir and os.path.exists(additional_nii_dir):
        print(f"\nProcessing additional NIfTI files from: {additional_nii_dir}")
        for file_name in os.listdir(additional_nii_dir):
            if file_name.endswith('.nii.gz') or file_name.endswith('.nii'):
                file_path = os.path.join(additional_nii_dir, file_name)
                try:
                    nifti_image = nib.load(file_path)
                    print(f"\n===== {file_name} =====")
                    print(f"File path: {file_path}")
                    print(f"Dimensions: {nifti_image.shape}")
                    print(f"Data type: {nifti_image.get_data_dtype()}")
                    print(f"Voxel size: {nifti_image.header.get_zooms()}")
                    
                    # Extract frame information
                    if len(nifti_image.shape) >= 4:
                        frames = nifti_image.shape[3]
                        print(f"Number of frames: {frames}")
                    else:
                        print("This file does not contain frame information")
                except Exception as e:
                    print(f"Failed to load additional NIfTI file: {e}")

if __name__ == "__main__":
    input_root_directory = r"Y:\home\jiayf\qunex\field\BDUA_001\BDUA_001"  # DICOM folder root directory
    output_directory = r"Y:\home\jiayf\qunex\field\BDUA_001\BDUA_001"  # Output directory
    additional_nii_directory = r"Y:\home\jiayf\qunex\test\sessions\BDUA_001\nii"  # ${STUDY_FOLDER}/sessions/BDUA_001/nii/5011.nii.gz (5011: restfMRI_mb7_240_iso2.5mm_SaveBySlc)

    convert_dicom_to_nifti(input_root_directory, output_directory, additional_nii_directory)

Part of the output obtained from the above code is as follows (I only kept the fmri information), it’s easy to see that the fmri data we manually formatted has 240 time points, while the fmri converted using the import_dicom format has only 48 time points.

Processing DICOM folders from: Y:\home\jiayf\qunex\field\BDUA_001\BDUA_001
Output NIfTI files will be saved to: Y:\home\jiayf\qunex\field\BDUA_001\BDUA_001

===== restfMRI_mb7_240_iso2.5mm_SaveBy_501 =====
Output file: Y:\home\jiayf\qunex\field\BDUA_001\BDUA_001\restfMRI_mb7_240_iso2.5mm_SaveBy_501.nii.gz
Dimensions: (84, 84, 63, 240)
Data type: uint16
Voxel size: (2.5, 2.5, 2.5, 2.0)
Number of frames: 240

Processing additional NIfTI files from: Y:\home\jiayf\qunex\test\sessions\BDUA_001\nii

===== 5011.nii.gz =====
File path: Y:\home\jiayf\qunex\test\sessions\BDUA_001\nii\5011.nii.gz
Dimensions: (84, 84, 63, 48)
Data type: uint16
Voxel size: (2.5, 2.5, 2.5, 2.0)
Number of frames: 48

It is indeed WEIRD that the same version of dcm2niix gives different results.

Best, YJia

Hi JYia,

I reviewed the outgoing dcm2niix calls and they are nearly identical to the one you execute manually. The only difference is that in our code, we also set -b y but that is the default anyway, so this should be irrelevant.

I think the only option here is that you upload one problematic sessions to a location where I can get it so I can investigate properly on my end. Is this possible or is the data too sensitive to share?

Best, Jure

Hi, Jure

Thank you very much for your help, I also think you need to use a case of test data to check what is going wrong.

I will provide you with an example of the data with the full scan sequence so that you can perhaps get an idea of the UIH field map data at the same time as checking the import_dicom problem. Of course, since this data is not publicly available at the moment, I would like you to remove the links to the data once you have downloaded it and, if possible, not share this example with others.

May I ask if you can receive the data from Google Drive? I will send you the Google Drive link privately later and unshare it after your next reply or after 24h.

Best, YJia

Hi JYia,

Sorry this is taking a bit longer, my development system broke down and I was not able to work on this. I got it back today and will try to get back to you by the end of this week.

Best, Jure