[RESOLVED] Importing new HCP data into existing QUNEX folder set up

Hi everyone,

We have on-boarded HCP diffusion data and pre-processed this data. We now want to onboard resting state HCP data into the same folder structure but are running into the errors with import_hcp.

The error seems to be is it wants to look for hcpls data in /share/HCP-Disease/DCAM/output/NDARAB462FH0_MR/sessions/MR/hcpls which doesn’t exist as the correct file path is /share/HCP-Disease/DCAM/output/NDARAB462FH0_MR/sessions/NDARAB462FH0_MR/hcpls

I have tried numerous time with changing the hcplsname. I can get it to work if I create_study and create a new folder structure, however we want it in the existing folder structure not a new one.

Does anyone know a fix for this? Or is there a different way to onboard new data into an existing set up?

Thanks so much!

Call:

qunex version 0.99.2

qunex_container import_hcp \
    --bind="/imgshare:/imgshare,/share/HCP-Disease/DCAM:/share/HCP-Disease/DCAM,/share/ndadata:/share/ndadata" \
    --sessionsfolder="/share/HCP-Disease/DCAM/output/NDARAB462FH0_MR/sessions" \
    --inbox="/share/HCP-Disease/DCAM/data/NDARAB462FH0" \
    --sessions="${SESSIONS}" \
    --action="copy" \
    --overwrite="no" \
    --archive="leave" \
    --nameformat="(?P<subject_id>[^/]+?)/unprocessed/(?P<session_name>.*?)/(?P<data>.*)" \
    --hcplsname="NDARAB462FH0_MR" \
  --container="${QUNEXCONIMAGE}"

Logs:

started running import_hcp at 2024-06-24 16:43:24, track progress in /share/HCP-Disease/DCAM/output/NDARAB462FH0_MR/processing/logs/comlogs/tmp_import_hcp_2024-06-24_16.43.24.055029.log
call: gmri import_hcp sessionsfolder="/share/HCP-Disease/DCAM/output/NDARAB462FH0_MR/sessions" inbox="/share/HCP-Disease/DCAM/data/NDARAB462FH0" action="copy" overwrite="no" archive="leave" nameformat="(?P<subject_id>[^/]+?)/unprocessed/(?P<session_name>.*?)/(?P<data>.*)" hcplsname="NDARAB462FH0_MR" sessions="NDARAB462FH0_MR"
-----------------------------------------
Running import_hcp
==================
--> identifying files in /share/HCP-Disease/DCAM/data/NDARAB462FH0
--> mapping files to QuNex hcpls folders
    --> incomplete hcpls for session MR already exists: cleaning session
info: subject MR
Running map_hcpls2nii for subject MR
====================================
===> WARNING:
     No hcpls folder present!
     There is no hcpls data in session folder /share/HCP-Disease/DCAM/output/NDARAB462FH0_MR/sessions/MR/hcpls
     Please import HCPLS data first!


Final report
============
subject MR failed

===> ERROR in completing import_hcp:
     Some actions failed
     Please check report!


-----------------------------------------
Finished at 2024-06-24 16:44:04

Hi!

Import functions do not support data appending (e.g. first importing structural data, then importing bolds, and finally importing diffusion). I think this could be useful to have, we will discuss this internally to see how much work this would be.

There are several workarounds about how do overcome this though.

  1. Create a new QuNex study, import all data into it with a single import command and then copy what you already processed (the hcp folder) within the session into the new study. The result will be a study that includes all your unprocessed data along with what you already processed.

  2. Create a new study and import the BOLD data into it. Copy that data into your existing study. Here you will have to also manually merge the session.txt file. Next, you amend the mapping file and rerun the prep functions (create_session_info, create_batch, setup_hcp). This one might be a bit tricky because image/sequence numbers might be in conflict.

Best, Jure

Thanks for getting back to me so quickly and thanks for the workarounds!

Would also creating a new qunex study then creating a symlink to the hcp folder be a workaround?

Thanks again

Yes, I believe that should work as well.