[RESOLVED] Error with importing pre-processed HCP-D data

leonmartin · February 1, 2024, 4:52pm

Hi
I want to process the diffusion data from the HCP-development dataset with the already preprocessed structural data. However, I failed trying to import the processed data with the import_hcp command and the new --processed_data flag (QuNex v0.99.2).

Do I miss some naming specifications or extra settings?

Many thanks in advance! <3

My command is:

qunex_container import_hcp --sessionsfolder=$STUDYDIR/sessions \
    --inbox=$RAWDIR/imagingcollection01 --action='copy' \
    --overwrite=yes --archive='leave' \
    --processed_data=$RAWDIR/fmriresults01

And runs with the following log:

# Generated by QuNex 0.99.2 on 2024-02-01_17.44.49.906323
#
call: gmri import_hcp sessionsfolder="/fast/scratch/users/martinl_c/dHCP_diffusion/sessions" inbox="/fast/scratch/users/martinl_c/raw_data/imagingcollection01" action="copy" overwrite="yes" archive="leave" processed_data="/fast/scratch/users/martinl_c/raw_data/fmriresults01"
-----------------------------------------
Running import_hcp
==================
--> identifying files in /fast/scratch/users/martinl_c/raw_data/imagingcollection01
--> mapping files to QuNex hcpls folders
ERROR: Could not parse file: /fast/scratch/users/martinl_c/raw_data/imagingcollection01/manifests/HCD0001305_V1_MR_UnprocTfmriCarit_manifest.json
ERROR: Could not parse file: /fast/scratch/users/martinl_c/raw_data/imagingcollection01/manifests/HCD0001305_V1_MR_UnprocRfmri_manifest.json
ERROR: Could not parse file: /fast/scratch/users/martinl_c/raw_data/imagingcollection01/manifests/HCD0001305_V1_MR_UnprocPcasl_manifest.json
ERROR: Could not parse file: /fast/scratch/users/martinl_c/raw_data/imagingcollection01/manifests/HCD0001305_V1_MR_UnprocTfmriGuessing_manifest.json
    --> hcpls for session HCD0001305_V1_MR already exists: cleaning session
info: subject HCD0001305, session V1_MR
Running map_hcpls2nii for subject HCD0001305, session V1_MR
===========================================================
--> filesort: name_type_se
--> cleaned nii folder, removed existing files
--> removed existing session.txt file
--> removed existing session_hcp.txt file
--> linked 01.nii.gz <-- HCD0001305_V1_MR_T1w_MPR_vNav_4e_e1e2_mean.nii.gz
--> linked 02.nii.gz <-- HCD0001305_V1_MR_T2w_SPC_vNav.nii.gz
--> linked 03.nii.gz <-- HCD0001305_V1_MR_tfMRI_CARIT_AP.nii.gz
--> linked 04.nii.gz <-- HCD0001305_V1_MR_tfMRI_CARIT_PA.nii.gz
--> linked 05.nii.gz <-- HCD0001305_V1_MR_dMRI_dir98_AP.nii.gz
==> WARNING: bval/bvec files were not found and were not mapped for 05.nii.gz!
==> ERROR: bval/bvec files were not found and were not mapped: 05.bval/.bvec <-- HCD0001305_V1_MR_dMRI_dir98_AP.bval/.bvec
--> linked 06.nii.gz <-- HCD0001305_V1_MR_dMRI_dir98_PA.nii.gz
==> WARNING: bval/bvec files were not found and were not mapped for 06.nii.gz!
==> ERROR: bval/bvec files were not found and were not mapped: 06.bval/.bvec <-- HCD0001305_V1_MR_dMRI_dir98_PA.bval/.bvec
--> linked 07.nii.gz <-- HCD0001305_V1_MR_dMRI_dir99_AP.nii.gz
==> WARNING: bval/bvec files were not found and were not mapped for 07.nii.gz!
==> ERROR: bval/bvec files were not found and were not mapped: 07.bval/.bvec <-- HCD0001305_V1_MR_dMRI_dir99_AP.bval/.bvec
--> linked 08.nii.gz <-- HCD0001305_V1_MR_dMRI_dir99_PA_SBRef.nii.gz
--> linked 09.nii.gz <-- HCD0001305_V1_MR_dMRI_dir99_PA.nii.gz
--> linked 10.nii.gz <-- HCD0001305_V1_MR_SpinEchoFieldMap1_AP.nii.gz
--> linked 11.nii.gz <-- HCD0001305_V1_MR_SpinEchoFieldMap1_PA.nii.gz
--> linked 12.nii.gz <-- HCD0001305_V1_MR_rfMRI_REST1_AP_SBRef.nii.gz
--> linked 13.nii.gz <-- HCD0001305_V1_MR_rfMRI_REST1_AP.nii.gz
--> linked 14.nii.gz <-- HCD0001305_V1_MR_rfMRI_REST1_PA.nii.gz
--> linked 15.nii.gz <-- HCD0001305_V1_MR_SpinEchoFieldMap2_AP.nii.gz
--> linked 16.nii.gz <-- HCD0001305_V1_MR_tfMRI_EMOTION_PA_SBRef.nii.gz
--> linked 17.nii.gz <-- HCD0001305_V1_MR_SpinEchoFieldMap2_PA.nii.gz
--> linked 18.nii.gz <-- HCD0001305_V1_MR_tfMRI_GUESSING_AP_SBRef.nii.gz
--> linked 19.nii.gz <-- HCD0001305_V1_MR_tfMRI_GUESSING_AP.nii.gz
--> linked 20.nii.gz <-- HCD0001305_V1_MR_tfMRI_GUESSING_PA.nii.gz
--> linked 21.nii.gz <-- HCD0001305_V1_MR_SpinEchoFieldMap3_AP.nii.gz
--> linked 22.nii.gz <-- HCD0001305_V1_MR_SpinEchoFieldMap3_PA.nii.gz
--> linked 23.nii.gz <-- HCD0001305_V1_MR_rfMRI_REST2_AP_SBRef.nii.gz
===> WARNING:
     Not all actions completed successfully!
     Some files for session HCD0001305_V1_MR were not mapped successfully!
     Please check logs and data!


--> copying processed data

Final report
============
subject HCD0001305, session V1_MR failed

===> ERROR in completing import_hcp:
     Some actions failed
     Please check report!


-----------------------------------------
Finished at 2024-02-01 17:45:02

demsarjure · February 2, 2024, 8:18am

Hi Leon,

1/ I assume that bval and bvec files are where they are supposed to be and named as they should be. That is inside HCD0001305_V1_MR/unprocessed/diffusion. For example, there you should have 4 files for each dMRI:

HCD0001305_V1_MR_dMRI_dir98_AP.nii.gz
HCD0001305_V1_MR_dMRI_dir98_AP.json
HCD0001305_V1_MR_dMRI_dir98_AP.bval
HCD0001305_V1_MR_dMRI_dir98_AP.bvec

2/ You could also try onboarding without the processed data. For example for a single session as a test:

qunex_container import_hcp \
    --sessionsfolder=$STUDYDIR/sessions \
    --sessions="HCD0001305_V1_MR" \
    --inbox=$RAWDIR/imagingcollection01 \
    --action='copy' \
    --overwrite=yes \
    --archive='leave' \
    --container=<container_path>

If this works and there are indeed some bugs I am not aware of is then a workaround that might work: you can simply copy the processed data manually into the QuNex study:

HCP_DIR=$STUDYDIR/sessions/HCD0001305_V1_MR/hcp/HCD0001305_V1_MR
mkdir -p $HCP_DIR
cp -R $RAWDIR/fmriresults01/HCD0001305_V1_MR $HCP_DIR

This should give you the processed folders (MNINonLinear, T1w, T2w) inside the $HCP_DIR. And you should be able to start running diffusion.

3/ If there is a bug, is there a way that you could zip one session both the unprocessed version inside $RAWDIR/imagingcollection01 and the processed one inside $RAWDIR/fmriresults01 and upload it somewhere where I can reach them? This will be by far for me the easiest to debug. I am unable to reproduce this on my end right now.

Best, Jure

leonmartin · February 12, 2024, 5:28pm

Dear Jure,

many thanks for your helpful answer! I managed to copy all the files and the data was read in well (below my session_hcp.txt file). However when I run

qunex_container hcp_diffusion --sessionsfolder="$STUDYDIR/sessions"\
    --sessions=$subjectid \
    --batchfile="$STUDYDIR/processing/batch.txt" --overwrite="yes" \
    --hcp_dwi_nogpu \
    --hcp_dwi_phasepos='PA' --hcp_dwi_phaseneg='AP'

I still get the following error:

Session id: HCD2322335_V1_MR 
[started on Monday, 12. February 2024 18:23:25]
Running HCP DiffusionPreprocessing Pipeline [HCPStyleData] ...
---> ERROR: No DWI files found, check the _hcp_dwi_phasepos and _hcp_dwi_phaseneg parameters.
---> ERROR: No pos direction files were found! Both images with pos and neg directions are required for hcp_diffusion. If you have data with only one direction, you can use dwi_legacy_gpu.
---> ERROR: Could not find gradient distortion coefficients file: NONE            ... DWI specific gradient distortion coefficients file or NONE..
---> Using image specific EchoSpacing: 0.6899980000
---> Session can not be processed.

am I still missing something?

I greatly appreciate your help, thank you a lot!

Best,
L

session_hcp.txt

# Generated by QuNex 0.99.2 on 2024-02-12_18.06.37.611281
#
session: HCD2322335_V1_MR
subject: HCD2322335
hcpfs: /fast/scratch/users/martinl_c/dHCP_diffusion/sessions/HCD2322335_V1_MR/hcpls
raw_data: /fast/scratch/users/martinl_c/dHCP_diffusion/sessions/HCD2322335_V1_MR/nii
hcp: /fast/scratch/users/martinl_c/dHCP_diffusion/sessions/HCD2322335_V1_MR/hcp

hcpready: true

institution: CMRR_3T-C
device: Siemens|Prisma|166007

01: DWIref:dir98_AP     : dMRI_dir98_AP_SBRef           : phenc(AP) : EchoSpacing(0.6899980000) : filename(dMRI_dir98_AP_SBRef)
02: DWI:dir98_AP        : dMRI_dir98_AP                 : phenc(AP) : EchoSpacing(0.6899980000) : filename(dMRI_dir98_AP)
03: DWIref:dir98_PA     : dMRI_dir98_PA_SBRef           : phenc(PA) : EchoSpacing(0.6899980000) : filename(dMRI_dir98_PA_SBRef)
04: DWI:dir98_PA        : dMRI_dir98_PA                 : phenc(PA) : EchoSpacing(0.6899980000) : filename(dMRI_dir98_PA)
05: DWIref:dir99_AP     : dMRI_dir99_AP_SBRef           : phenc(AP) : EchoSpacing(0.6899980000) : filename(dMRI_dir99_AP_SBRef)
06: DWI:dir99_AP        : dMRI_dir99_AP                 : phenc(AP) : EchoSpacing(0.6899980000) : filename(dMRI_dir99_AP)
07: DWIref:dir99_PA     : dMRI_dir99_PA_SBRef           : phenc(PA) : EchoSpacing(0.6899980000) : filename(dMRI_dir99_PA_SBRef)
08: DWI:dir99_PA        : dMRI_dir99_PA                 : phenc(PA) : EchoSpacing(0.6899980000) : filename(dMRI_dir99_PA)

demsarjure · February 13, 2024, 8:51am

One possibility could be that the folder with the study is not visible to the system in the container, try adding the bind parameter to the qunex_container call.

...
--bind=/fast/scratch/users/martinl_c:/fast/scratch/users/martinl_c
...

Also, please provide the full qunex_container call, the container parameter is also missing in your example, just so I know whether you are using Singularity or docker.

demsarjure · February 27, 2024, 2:59pm

Hi Leon,

Did we manage to help you out on this, or do you still have any questions or issues that we can help you with?

Best, Jure