[RESOLVED] Error with importing pre-processed HCP-D data

Hi :slight_smile:
I want to process the diffusion data from the HCP-development dataset with the already preprocessed structural data. However, I failed trying to import the processed data with the import_hcp command and the new --processed_data flag (QuNex v0.99.2).

Do I miss some naming specifications or extra settings?

Many thanks in advance! <3

My command is:

qunex_container import_hcp --sessionsfolder=$STUDYDIR/sessions \
    --inbox=$RAWDIR/imagingcollection01 --action='copy' \
    --overwrite=yes --archive='leave' \
    --processed_data=$RAWDIR/fmriresults01

And runs with the following log:

# Generated by QuNex 0.99.2 on 2024-02-01_17.44.49.906323
#
call: gmri import_hcp sessionsfolder="/fast/scratch/users/martinl_c/dHCP_diffusion/sessions" inbox="/fast/scratch/users/martinl_c/raw_data/imagingcollection01" action="copy" overwrite="yes" archive="leave" processed_data="/fast/scratch/users/martinl_c/raw_data/fmriresults01"
-----------------------------------------
Running import_hcp
==================
--> identifying files in /fast/scratch/users/martinl_c/raw_data/imagingcollection01
--> mapping files to QuNex hcpls folders
ERROR: Could not parse file: /fast/scratch/users/martinl_c/raw_data/imagingcollection01/manifests/HCD0001305_V1_MR_UnprocTfmriCarit_manifest.json
ERROR: Could not parse file: /fast/scratch/users/martinl_c/raw_data/imagingcollection01/manifests/HCD0001305_V1_MR_UnprocRfmri_manifest.json
ERROR: Could not parse file: /fast/scratch/users/martinl_c/raw_data/imagingcollection01/manifests/HCD0001305_V1_MR_UnprocPcasl_manifest.json
ERROR: Could not parse file: /fast/scratch/users/martinl_c/raw_data/imagingcollection01/manifests/HCD0001305_V1_MR_UnprocTfmriGuessing_manifest.json
    --> hcpls for session HCD0001305_V1_MR already exists: cleaning session
info: subject HCD0001305, session V1_MR
Running map_hcpls2nii for subject HCD0001305, session V1_MR
===========================================================
--> filesort: name_type_se
--> cleaned nii folder, removed existing files
--> removed existing session.txt file
--> removed existing session_hcp.txt file
--> linked 01.nii.gz <-- HCD0001305_V1_MR_T1w_MPR_vNav_4e_e1e2_mean.nii.gz
--> linked 02.nii.gz <-- HCD0001305_V1_MR_T2w_SPC_vNav.nii.gz
--> linked 03.nii.gz <-- HCD0001305_V1_MR_tfMRI_CARIT_AP.nii.gz
--> linked 04.nii.gz <-- HCD0001305_V1_MR_tfMRI_CARIT_PA.nii.gz
--> linked 05.nii.gz <-- HCD0001305_V1_MR_dMRI_dir98_AP.nii.gz
==> WARNING: bval/bvec files were not found and were not mapped for 05.nii.gz!
==> ERROR: bval/bvec files were not found and were not mapped: 05.bval/.bvec <-- HCD0001305_V1_MR_dMRI_dir98_AP.bval/.bvec
--> linked 06.nii.gz <-- HCD0001305_V1_MR_dMRI_dir98_PA.nii.gz
==> WARNING: bval/bvec files were not found and were not mapped for 06.nii.gz!
==> ERROR: bval/bvec files were not found and were not mapped: 06.bval/.bvec <-- HCD0001305_V1_MR_dMRI_dir98_PA.bval/.bvec
--> linked 07.nii.gz <-- HCD0001305_V1_MR_dMRI_dir99_AP.nii.gz
==> WARNING: bval/bvec files were not found and were not mapped for 07.nii.gz!
==> ERROR: bval/bvec files were not found and were not mapped: 07.bval/.bvec <-- HCD0001305_V1_MR_dMRI_dir99_AP.bval/.bvec
--> linked 08.nii.gz <-- HCD0001305_V1_MR_dMRI_dir99_PA_SBRef.nii.gz
--> linked 09.nii.gz <-- HCD0001305_V1_MR_dMRI_dir99_PA.nii.gz
--> linked 10.nii.gz <-- HCD0001305_V1_MR_SpinEchoFieldMap1_AP.nii.gz
--> linked 11.nii.gz <-- HCD0001305_V1_MR_SpinEchoFieldMap1_PA.nii.gz
--> linked 12.nii.gz <-- HCD0001305_V1_MR_rfMRI_REST1_AP_SBRef.nii.gz
--> linked 13.nii.gz <-- HCD0001305_V1_MR_rfMRI_REST1_AP.nii.gz
--> linked 14.nii.gz <-- HCD0001305_V1_MR_rfMRI_REST1_PA.nii.gz
--> linked 15.nii.gz <-- HCD0001305_V1_MR_SpinEchoFieldMap2_AP.nii.gz
--> linked 16.nii.gz <-- HCD0001305_V1_MR_tfMRI_EMOTION_PA_SBRef.nii.gz
--> linked 17.nii.gz <-- HCD0001305_V1_MR_SpinEchoFieldMap2_PA.nii.gz
--> linked 18.nii.gz <-- HCD0001305_V1_MR_tfMRI_GUESSING_AP_SBRef.nii.gz
--> linked 19.nii.gz <-- HCD0001305_V1_MR_tfMRI_GUESSING_AP.nii.gz
--> linked 20.nii.gz <-- HCD0001305_V1_MR_tfMRI_GUESSING_PA.nii.gz
--> linked 21.nii.gz <-- HCD0001305_V1_MR_SpinEchoFieldMap3_AP.nii.gz
--> linked 22.nii.gz <-- HCD0001305_V1_MR_SpinEchoFieldMap3_PA.nii.gz
--> linked 23.nii.gz <-- HCD0001305_V1_MR_rfMRI_REST2_AP_SBRef.nii.gz
===> WARNING:
     Not all actions completed successfully!
     Some files for session HCD0001305_V1_MR were not mapped successfully!
     Please check logs and data!


--> copying processed data

Final report
============
subject HCD0001305, session V1_MR failed

===> ERROR in completing import_hcp:
     Some actions failed
     Please check report!


-----------------------------------------
Finished at 2024-02-01 17:45:02

Hi Leon,

1/ I assume that bval and bvec files are where they are supposed to be and named as they should be. That is inside HCD0001305_V1_MR/unprocessed/diffusion. For example, there you should have 4 files for each dMRI:

HCD0001305_V1_MR_dMRI_dir98_AP.nii.gz
HCD0001305_V1_MR_dMRI_dir98_AP.json
HCD0001305_V1_MR_dMRI_dir98_AP.bval
HCD0001305_V1_MR_dMRI_dir98_AP.bvec

2/ You could also try onboarding without the processed data. For example for a single session as a test:

qunex_container import_hcp \
    --sessionsfolder=$STUDYDIR/sessions \
    --sessions="HCD0001305_V1_MR" \
    --inbox=$RAWDIR/imagingcollection01 \
    --action='copy' \
    --overwrite=yes \
    --archive='leave' \
    --container=<container_path>

If this works and there are indeed some bugs I am not aware of is then a workaround that might work: you can simply copy the processed data manually into the QuNex study:

HCP_DIR=$STUDYDIR/sessions/HCD0001305_V1_MR/hcp/HCD0001305_V1_MR
mkdir -p $HCP_DIR
cp -R $RAWDIR/fmriresults01/HCD0001305_V1_MR $HCP_DIR

This should give you the processed folders (MNINonLinear, T1w, T2w) inside the $HCP_DIR. And you should be able to start running diffusion.


3/ If there is a bug, is there a way that you could zip one session both the unprocessed version inside $RAWDIR/imagingcollection01 and the processed one inside $RAWDIR/fmriresults01 and upload it somewhere where I can reach them? This will be by far for me the easiest to debug. I am unable to reproduce this on my end right now.

Best, Jure

1 Like

Dear Jure,

many thanks for your helpful answer! I managed to copy all the files and the data was read in well (below my session_hcp.txt file). However when I run

qunex_container hcp_diffusion --sessionsfolder="$STUDYDIR/sessions"\
    --sessions=$subjectid \
    --batchfile="$STUDYDIR/processing/batch.txt" --overwrite="yes" \
    --hcp_dwi_nogpu \
    --hcp_dwi_phasepos='PA' --hcp_dwi_phaseneg='AP'

I still get the following error:

Session id: HCD2322335_V1_MR 
[started on Monday, 12. February 2024 18:23:25]
Running HCP DiffusionPreprocessing Pipeline [HCPStyleData] ...
---> ERROR: No DWI files found, check the _hcp_dwi_phasepos and _hcp_dwi_phaseneg parameters.
---> ERROR: No pos direction files were found! Both images with pos and neg directions are required for hcp_diffusion. If you have data with only one direction, you can use dwi_legacy_gpu.
---> ERROR: Could not find gradient distortion coefficients file: NONE            ... DWI specific gradient distortion coefficients file or NONE..
---> Using image specific EchoSpacing: 0.6899980000
---> Session can not be processed.

am I still missing something?

I greatly appreciate your help, thank you a lot!

Best,
L

session_hcp.txt

# Generated by QuNex 0.99.2 on 2024-02-12_18.06.37.611281
#
session: HCD2322335_V1_MR
subject: HCD2322335
hcpfs: /fast/scratch/users/martinl_c/dHCP_diffusion/sessions/HCD2322335_V1_MR/hcpls
raw_data: /fast/scratch/users/martinl_c/dHCP_diffusion/sessions/HCD2322335_V1_MR/nii
hcp: /fast/scratch/users/martinl_c/dHCP_diffusion/sessions/HCD2322335_V1_MR/hcp

hcpready: true

institution: CMRR_3T-C
device: Siemens|Prisma|166007

01: DWIref:dir98_AP     : dMRI_dir98_AP_SBRef           : phenc(AP) : EchoSpacing(0.6899980000) : filename(dMRI_dir98_AP_SBRef)
02: DWI:dir98_AP        : dMRI_dir98_AP                 : phenc(AP) : EchoSpacing(0.6899980000) : filename(dMRI_dir98_AP)
03: DWIref:dir98_PA     : dMRI_dir98_PA_SBRef           : phenc(PA) : EchoSpacing(0.6899980000) : filename(dMRI_dir98_PA_SBRef)
04: DWI:dir98_PA        : dMRI_dir98_PA                 : phenc(PA) : EchoSpacing(0.6899980000) : filename(dMRI_dir98_PA)
05: DWIref:dir99_AP     : dMRI_dir99_AP_SBRef           : phenc(AP) : EchoSpacing(0.6899980000) : filename(dMRI_dir99_AP_SBRef)
06: DWI:dir99_AP        : dMRI_dir99_AP                 : phenc(AP) : EchoSpacing(0.6899980000) : filename(dMRI_dir99_AP)
07: DWIref:dir99_PA     : dMRI_dir99_PA_SBRef           : phenc(PA) : EchoSpacing(0.6899980000) : filename(dMRI_dir99_PA_SBRef)
08: DWI:dir99_PA        : dMRI_dir99_PA                 : phenc(PA) : EchoSpacing(0.6899980000) : filename(dMRI_dir99_PA)

One possibility could be that the folder with the study is not visible to the system in the container, try adding the bind parameter to the qunex_container call.

...
--bind=/fast/scratch/users/martinl_c:/fast/scratch/users/martinl_c
...

Also, please provide the full qunex_container call, the container parameter is also missing in your example, just so I know whether you are using Singularity or docker.

Hi Leon,

Did we manage to help you out on this, or do you still have any questions or issues that we can help you with?

Best, Jure