[RESOLVED] Mismatch in image and host platforms and issue with import_hcp

Hi everyone,

I have followed the quickstart documentation to pull the docker container image (0.100.0) to my Mac M3 pc. I ran the create_study command successfully. Although I got the warning that the requested image’s platform(linux/amd64) does not match the host platform(linux/arm64/v8). I checked that Rosetta is installed in my machine. Can this mismatch create any problem later on? If yes, what is the workaround? Next when I ran the command import_hcp, I got an error that the extra argument masterinbox is invalid. I have, for the time being, downloaded one subject’s unprocessed data from NIMH data archive. I have tried running qunex import_hcp --h but it’s not identifying the command.

Below I have given the main portions from the logs. Please advice.

Thanks,
Mona

Call:

python3 qunex_container create_study
–studyfolder=“${STUDY_FOLDER}”
–container=“${QUNEX_CONTAINER”

python3 qunex_container import_hcp
–sessionsfolder=“${STUDY_FOLDER}/sessions”
–sessions=“${SESSIONS}”
–masterinbox=“${RAW_DATA}”
–archive=“leave”
–container=“${QUNEX_CONTAINER}”

Logs:

WARNING: The requested image’s platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

—> unsetting the following environment variables: PATH MATLABPATH PYTHONPATH QUNEXVer TOOLS QUNEXREPO QUNEXPATH QUNEXEXTENSIONS QUNEXLIBRARY QUNEXLIBRARYETC TemplateFolder FSL_FIXDIR FREESURFERDIR FREESURFER_HOME FREESURFER_SCHEDULER FreeSurferSchedulerDIR WORKBENCHDIR DCMNIIDIR DICMNIIDIR MATLABDIR MATLABBINDIR OCTAVEDIR OCTAVEPKGDIR OCTAVEBINDIR RDIR HCPWBDIR AFNIDIR PYLIBDIR FSLDIR FSLBINDIR PALMDIR QUNEXMCOMMAND HCPPIPEDIR CARET7DIR GRADUNWARPDIR HCPPIPEDIR_Templates HCPPIPEDIR_Bin HCPPIPEDIR_Config HCPPIPEDIR_PreFS HCPPIPEDIR_FS HCPPIPEDIR_PostFS HCPPIPEDIR_fMRISurf HCPPIPEDIR_fMRIVol HCPPIPEDIR_tfMRI HCPPIPEDIR_dMRI HCPPIPEDIR_dMRITract HCPPIPEDIR_Global HCPPIPEDIR_tfMRIAnalysis HCPCIFTIRWDIR MSMBin HCPPIPEDIR_dMRITractFull HCPPIPEDIR_dMRILegacy AutoPtxFolder EDDYCUDA USEOCTAVE QUNEXENV CONDADIR MSMBINDIR MSMCONFIGDIR R_LIBS FSL_FIX_CIFTIRW FSFAST_HOME SUBJECTS_DIR MINC_BIN_DIR MNI_DIR MINC_LIB_DIR MNI_DATAPATH FSF_OUTPUT_FORMAT ANTSDIR CUDIMOT

— Full QuNex call for command: import_hcp

qunex import_hcp --sessionsfolder=“/Users/monamunsi/softwares/qunex/quickstart/HCP_EP/1001_01/sessions” --masterinbox=“/Users/monamunsi/softwares/qunex/data/HCP_EP” --archive=“leave” --sessions=“HCPEP_1001_01”


ERROR: Extra argument masterinbox is not valid! Please check your command!

ERROR in running import_hcp at gmri:
Invalid arguments
The extra argument(s) provided is/are not valid! [masterinbox]

The call received was:
(please note that when run through scheduler, all possible parameters,
even non relevant ones are passed)

qunex import_hcp
–sessionsfolder=/Users/monamunsi/softwares/qunex/quickstart/HCP_EP/1001_01/sessions
–masterinbox=/Users/monamunsi/softwares/qunex/data/HCP_EP
–archive=leave
–sessions=HCPEP_1001_01

Hi Mona,

Welcome to QuNex forums!

  1. Usually the mismatch results only in the warning and it works fine later on. We are not compiling the container for the newer Mac architecture specifically because the container is mainly used on compute servers and not so much on laptops. Keep me posted if there are any issues with this down the line.

  2. To get help, you also need to use the qunex_container call. So, to get help for import_hcp you can use:

qunex_container import_hcp --help \
    --container="${QUNEX_CONTAINER}"

Note that you can probably omit the python3 at the beginning as usually python3 is the default one. All help is also available at Background — QuNex documentation so you can get the same info in an easier-to-read format there.

  1. For import_hcp this is import_hcp — QuNex documentation. There is a bit of an inconsistency here on our end as the import_dicom command has a parameter called --masterinbox while import_hcp uses --inbox. Try swapping that.

  2. Also, when making posts, you can encapsulate code blocks and log excerpts with triple back ticks (repeat ` 3 times to start the block and the same to end it), this way it is much easier to read. For example, we get:

python3 qunex_container create_study
–studyfolder=“${STUDY_FOLDER}”
–container=“${QUNEX_CONTAINER”

instead of

python3 qunex_container create_study
–studyfolder=“${STUDY_FOLDER}”
–container=“${QUNEX_CONTAINER”

Hi,

Thank you for taking the time to respond to my questions.

  1. When I tried running qunex_container script without python3, it gave the error that zsh could not find qunex_container. But it ran when I added python3. Not sure what’s going on.

  2. I made the change from masterinbox to inbox and reran the import_hcp. It threw the error that no files were found in the data folder. Below are the calls and logs.

  3. I am also attaching the folder structure for your reference.

Calls:

    --sessionsfolder="${STUDY_FOLDER}/sessions" \
    --sessions="${SESSIONS}" \
    --inbox="${RAW_DATA}" \ 
    --archive="leave" \
    --container="${QUNEX_CONTAINER}" ```

Logs:

``` --- Full QuNex call for command: import_hcp

qunex import_hcp --sessionsfolder="/Users/monamunsi/softwares/qunex/quickstart//HCP_EP/1001_01//sessions" --inbox="/Users/monamunsi/softwares/qunex/data/HCP_EP" --archive="leave" --sessions="HCPEP_1001_01"

---------------------------------------------------------


started running import_hcp at 2024-08-01 17:26:10, track progress in /Users/monamunsi/softwares/qunex/quickstart/HCP_EP/1001_01/processing/logs/comlogs/tmp_import_hcp_2024-08-01_17.26.10.674739.log
call: gmri import_hcp sessionsfolder="/Users/monamunsi/softwares/qunex/quickstart//HCP_EP/1001_01//sessions" inbox="/Users/monamunsi/softwares/qunex/data/HCP_EP" archive="leave" sessions="HCPEP_1001_01"
-----------------------------------------
Running import_hcp
==================
---> identifying files in /Users/monamunsi/softwares/qunex/data/HCP_EP

---> ERROR in completing import_hcp:
     No files found
     No files were found to be processed at the specified inbox [/Users/monamunsi/softwares/qunex/data/HCP_EP]!
     Please check your path! ```

Folder structure:

``` .
├── data
│   ├── HCP_EP
│   │   └── 1001_01_MR.zip
│   └── qunex_container
├── quickstart
│   └── HCP_EP
│       └── 1001_01
│           ├── analysis
│           │   └── scripts
│           ├── info
│           │   ├── bids
│           │   ├── demographics
│           │   ├── hcpls
│           │   ├── stimuli
│           │   └── tasks
│           ├── processing
│           │   ├── lists
│           │   ├── logs
│           │   │   ├── batchlogs
│           │   │   │   ├── qunex_container_command_2024-07-31_22.35.18.781360.sh
│           │   │   │   └── qunex_container_command_2024-08-01_13.05.34.392649.sh
│           │   │   ├── comlogs
│           │   │   │   ├── done_create_study_2024-08-01_02.33.51.727370.log
│           │   │   │   ├── done_create_study_2024-08-01_17.04.52.559242.log
│           │   │   │   └── error_import_hcp_2024-08-01_17.05.36.420259.log
│           │   │   ├── runchecks
│           │   │   └── runlogs
│           │   │       ├── Log-create_study_2024-08-01_02.33.51.727370.log
│           │   │       ├── Log-create_study_2024-08-01_17.04.52.559242.log
│           │   │       └── Log-import_hcp_2024-08-01_17.05.36.420259.log
│           │   ├── scenes
│           │   │   └── QC
│           │   │       ├── BOLD
│           │   │       ├── DWI
│           │   │       ├── T1w
│           │   │       ├── T2w
│           │   │       └── myelin
│           │   └── scripts
│           └── sessions
│               ├── QC
│               ├── archive
│               │   ├── BIDS
│               │   ├── EEG
│               │   ├── HCPLS
│               │   ├── MR
│               │   └── behavior
│               ├── inbox
│               │   ├── BIDS
│               │   ├── EEG
│               │   ├── HCPLS
│               │   ├── MR
│               │   ├── behavior
│               │   ├── concs
│               │   └── events
│               └── specs
│                   └── parameters_example.txt
└── qunex_container

Thanks,
Mona

Your post is truncated a bit on the top so I am doing some guesswork here. Based on the folder structure I think the settings for import_hcp should be:

STUDY_FOLDER="/quickstart/HCP_EP/1001_01"
SESSIONS="1001_01_MR"
RAW_DATA="/data/HCP_EP"

Jure

Hello,

The import_hcp worked after I made the suggested changes. However, when I tried to run create_session_info, I was getting errors for the mapping file. A little bit of digging around showed that the session.txt file thus generated doesn’t have all the rest bolds. The unprocessed dataset has the following files: rfMRI_REST1_AP rfMRI_REST1_PA rfMRI_REST2_AP rfMRI_REST2_PA T1w_MPR T2w_SPC.

When I run import_hcp, it’s not mapping all the nifti files. I’m going to put everything in the Calls portion. I am pasting the session.txt file as well. Basically, import_hcp is skipping a lot of files and I would appreciate if you could help me fixing it. I am also including the folder structure.

Calls:

qunex import_hcp --sessionsfolder="/sc/arion/projects/WenglerLab_SczEnsembles//qunex/quickstart/HCP_EP/1001_01/sessions" --inbox="/sc/arion/projects/WenglerLab_SczEnsembles//data/HCP_EP/Package_1231451/imagingcollection01" --archive="leave" --sessions="1001_01_MR"


Logs:

call: gmri import_hcp sessionsfolder="/sc/arion/projects/WenglerLab_SczEnsembles//qunex/quickstart/HCP_EP/1001_01/sessions" inbox="/sc/arion/projects/WenglerLab_SczEnsembles//data/HCP_EP/Package_1231451/imagingcollection01" archive="leave" sessions="1001_01_MR"
-----------------------------------------
Running import_hcp
==================
---> creating inbox HCPLS folder
---> creating archive HCPLS folder
---> identifying files in /sc/arion/projects/WenglerLab_SczEnsembles//data/HCP_EP/Package_1231451/imagingcollection01
---> mapping files to QuNex hcpls folders
    ---> creating hcpl session 1001_01_MR
info: subject 1001, session 01_MR
Running map_hcpls2nii for subject 1001, session 01_MR
=====================================================
---> filesort: name_type_se
---> linked 01.nii.gz <-- 1001_01_MR_T1w_MPR.nii.gz
---> linked 02.nii.gz <-- 1001_01_MR_T2w_SPC.nii.gz
---> linked 03.nii.gz <-- 1001_01_MR_SpinEchoFieldMap1_AP.nii.gz
---> linked 04.nii.gz <-- 1001_01_MR_SpinEchoFieldMap1_PA.nii.gz
---> linked 05.nii.gz <-- 1001_01_MR_rfMRI_REST1_AP_SBRef.nii.gz
---> linked 06.nii.gz <-- 1001_01_MR_rfMRI_REST1_AP.nii.gz
---> linked 07.nii.gz <-- 1001_01_MR_rfMRI_REST1_PA_SBRef.nii.gz
---> linked 08.nii.gz <-- 1001_01_MR_SpinEchoFieldMap2_AP.nii.gz
---> linked 09.nii.gz <-- 1001_01_MR_SpinEchoFieldMap2_PA.nii.gz
---> linked 10.nii.gz <-- 1001_01_MR_rfMRI_REST2_AP_SBRef.nii.gz
---> linked 11.nii.gz <-- 1001_01_MR_rfMRI_REST2_PA_SBRef.nii.gz

Final report
============
subject 1001, session 01_MR completed ok. 11 images mapped ```

session.txt:

```01: T1w                 : T1w_MPR                       : DwellTime(0.0000071000) : UnwarpDir(z) : filename(T1w_MPR)
02: T2w                 : T2w_SPC                       : DwellTime(0.0000021000) : UnwarpDir(z) : filename(T2w_SPC)
03: SE-FM-AP            : SpinEchoFieldMap_AP           : se(1) : phenc(AP) : EchoSpacing(0.0005800090) : filename(SpinEchoFieldMap_AP)
04: SE-FM-PA            : SpinEchoFieldMap_PA           : se(1) : phenc(PA) : EchoSpacing(0.0005800090) : filename(SpinEchoFieldMap_PA)
05: boldref1:REST1	: rfMRI_REST1_AP_SBRef          : se(1) : phenc(AP) : EchoSpacing(0.0005800090) : filename(rfMRI_REST1_AP_SBRef)
06: bold1:REST1         : rfMRI_REST1_AP                : se(1) : phenc(AP) : EchoSpacing(0.0005800090) : filename(rfMRI_REST1_AP)
07: boldref2:REST1	: rfMRI_REST1_PA_SBRef          : se(1) : phenc(PA) : EchoSpacing(0.0005800090) : filename(rfMRI_REST1_PA_SBRef)
08: SE-FM-AP            : SpinEchoFieldMap_AP           : se(2) : phenc(AP) : EchoSpacing(0.0005800090) : filename(SpinEchoFieldMap_AP)
09: SE-FM-PA            : SpinEchoFieldMap_PA           : se(2) : phenc(PA) : EchoSpacing(0.0005800090) : filename(SpinEchoFieldMap_PA)
10: boldref3:REST2	: rfMRI_REST2_AP_SBRef          : se(2) : phenc(AP) : EchoSpacing(0.0005800090) : filename(rfMRI_REST2_AP_SBRef)
11: boldref4:REST2	: rfMRI_REST2_PA_SBRef          : se(2) : phenc(PA) : EchoSpacing(0.0005800090) : filename(rfMRI_REST2_PA_SBRef)

Folder structure:


RAW_DATA="/sc/arion/projects/WenglerLab_SczEnsembles//data/HCP_EP/Package_1231451/imagingcollection01"

SESSIONS="1001_01_MR"

file:///sc/arion/projects/WenglerLab_SczEnsembles/qunex
|-- archive
|   `-- HCPLS
|-- comlogs
|   |-- error_import_hcp_2024-08-12_15.22.32.750548.log
|   |-- error_import_hcp_2024-08-12_15.22.52.636098.log
|   `-- error_import_hcp_2024-08-12_15.24.14.945316.log
|-- inbox
|   `-- HCPLS
|-- quickstart
|   `-- HCP_EP
|       `-- 1001_01
|           |-- analysis
|           |   `-- scripts
|           |-- info
|           |   |-- bids
|           |   |-- demographics
|           |   |-- hcpls
|           |   |   `-- parameters.txt
|           |   |-- stimuli
|           |   `-- tasks
|           |-- processing
|           |   |-- lists
|           |   |-- logs
|           |   |   |-- batchlogs
|           |   |   |   `-- qunex_container_command_2024-08-13_12.00.30.564621.sh
|           |   |   |-- comlogs
|           |   |   |   |-- done_create_study_2024-08-13_11.59.50.628502.log
|           |   |   |   `-- done_import_hcp_2024-08-13_12.00.36.552664.log
|           |   |   |-- runchecks
|           |   |   `-- runlogs
|           |   |       |-- Log-create_study_2024-08-13_11.59.50.628502.log
|           |   |       `-- Log-import_hcp_2024-08-13_12.00.36.552664.log
|           |   |-- scenes
|           |   |   `-- QC
|           |   |       |-- BOLD
|           |   |       |-- DWI
|           |   |       |-- T1w
|           |   |       |-- T2w
|           |   |       `-- myelin
|           |   `-- scripts
|           `-- sessions
|               |-- 1001_01_MR
|               |   |-- hcpls
|               |   |   |-- T1w_MPR
|               |   |   |   |-- 1001_01_MR_T1w_MPR.json
|               |   |   |   |-- 1001_01_MR_T1w_MPR.nii.gz
|               |   |   |   `-- OTHER_FILES
|               |   |   |       |-- 1001_01_MR_BIAS_32CH.json
|               |   |   |       |-- 1001_01_MR_BIAS_32CH.nii.gz
|               |   |   |       |-- 1001_01_MR_BIAS_BC.json
|               |   |   |       |-- 1001_01_MR_BIAS_BC.nii.gz
|               |   |   |       |-- 1001_01_MR_SpinEchoFieldMap1_AP.json
|               |   |   |       |-- 1001_01_MR_SpinEchoFieldMap1_AP.nii.gz
|               |   |   |       |-- 1001_01_MR_SpinEchoFieldMap1_PA.json
|               |   |   |       |-- 1001_01_MR_SpinEchoFieldMap1_PA.nii.gz
|               |   |   |       |-- 1001_01_MR_T1w_MPR_Norm.json
|               |   |   |       |-- 1001_01_MR_T1w_MPR_Norm.nii.gz
|               |   |   |       `-- session_report.csv
|               |   |   |-- T2w_SPC
|               |   |   |   |-- 1001_01_MR_T2w_SPC.json
|               |   |   |   |-- 1001_01_MR_T2w_SPC.nii.gz
|               |   |   |   `-- OTHER_FILES
|               |   |   |       |-- 1001_01_MR_BIAS_32CH.json
|               |   |   |       |-- 1001_01_MR_BIAS_32CH.nii.gz
|               |   |   |       |-- 1001_01_MR_BIAS_BC.json
|               |   |   |       |-- 1001_01_MR_BIAS_BC.nii.gz
|               |   |   |       |-- 1001_01_MR_SpinEchoFieldMap1_AP.json
|               |   |   |       |-- 1001_01_MR_SpinEchoFieldMap1_AP.nii.gz
|               |   |   |       |-- 1001_01_MR_SpinEchoFieldMap1_PA.json
|               |   |   |       |-- 1001_01_MR_SpinEchoFieldMap1_PA.nii.gz
|               |   |   |       |-- 1001_01_MR_T2w_SPC_Norm.json
|               |   |   |       `-- 1001_01_MR_T2w_SPC_Norm.nii.gz
|               |   |   |-- hcpls2nii.log
|               |   |   |-- rfMRI_REST1_AP
|               |   |   |   |-- 1001_01_MR_SpinEchoFieldMap1_AP.json
|               |   |   |   |-- 1001_01_MR_SpinEchoFieldMap1_AP.nii.gz
|               |   |   |   |-- 1001_01_MR_SpinEchoFieldMap1_PA.json
|               |   |   |   |-- 1001_01_MR_SpinEchoFieldMap1_PA.nii.gz
|               |   |   |   |-- 1001_01_MR_rfMRI_REST1_AP.json
|               |   |   |   |-- 1001_01_MR_rfMRI_REST1_AP.nii.gz
|               |   |   |   |-- 1001_01_MR_rfMRI_REST1_AP_InitialFrames.nii.gz
|               |   |   |   |-- 1001_01_MR_rfMRI_REST1_AP_SBRef.json
|               |   |   |   |-- 1001_01_MR_rfMRI_REST1_AP_SBRef.nii.gz
|               |   |   |   `-- OTHER_FILES
|               |   |   |       |-- 1001_01_MR_BIAS_32CH.json
|               |   |   |       |-- 1001_01_MR_BIAS_32CH.nii.gz
|               |   |   |       |-- 1001_01_MR_BIAS_BC.json
|               |   |   |       `-- 1001_01_MR_BIAS_BC.nii.gz
|               |   |   |-- rfMRI_REST1_PA
|               |   |   |   |-- 1001_01_MR_SpinEchoFieldMap1_AP.json
|               |   |   |   |-- 1001_01_MR_SpinEchoFieldMap1_AP.nii.gz
|               |   |   |   |-- 1001_01_MR_SpinEchoFieldMap1_PA.json
|               |   |   |   |-- 1001_01_MR_SpinEchoFieldMap1_PA.nii.gz
|               |   |   |   |-- 1001_01_MR_rfMRI_REST1_PA.json
|               |   |   |   |-- 1001_01_MR_rfMRI_REST1_PA.nii.gz.partial
|               |   |   |   |-- 1001_01_MR_rfMRI_REST1_PA_InitialFrames.nii.gz
|               |   |   |   |-- 1001_01_MR_rfMRI_REST1_PA_SBRef.json
|               |   |   |   |-- 1001_01_MR_rfMRI_REST1_PA_SBRef.nii.gz
|               |   |   |   `-- OTHER_FILES
|               |   |   |       |-- 1001_01_MR_BIAS_32CH.json
|               |   |   |       |-- 1001_01_MR_BIAS_32CH.nii.gz
|               |   |   |       |-- 1001_01_MR_BIAS_BC.json
|               |   |   |       `-- 1001_01_MR_BIAS_BC.nii.gz
|               |   |   |-- rfMRI_REST2_AP
|               |   |   |   |-- 1001_01_MR_SpinEchoFieldMap2_AP.json
|               |   |   |   |-- 1001_01_MR_SpinEchoFieldMap2_AP.nii.gz
|               |   |   |   |-- 1001_01_MR_SpinEchoFieldMap2_PA.json
|               |   |   |   |-- 1001_01_MR_SpinEchoFieldMap2_PA.nii.gz
|               |   |   |   |-- 1001_01_MR_rfMRI_REST2_AP.json
|               |   |   |   |-- 1001_01_MR_rfMRI_REST2_AP.nii.gz.partial
|               |   |   |   |-- 1001_01_MR_rfMRI_REST2_AP_InitialFrames.nii.gz
|               |   |   |   |-- 1001_01_MR_rfMRI_REST2_AP_SBRef.json
|               |   |   |   |-- 1001_01_MR_rfMRI_REST2_AP_SBRef.nii.gz
|               |   |   |   `-- OTHER_FILES
|               |   |   |       |-- 1001_01_MR_BIAS_32CH.json
|               |   |   |       |-- 1001_01_MR_BIAS_32CH.nii.gz
|               |   |   |       |-- 1001_01_MR_BIAS_BC.json
|               |   |   |       `-- 1001_01_MR_BIAS_BC.nii.gz
|               |   |   `-- rfMRI_REST2_PA
|               |   |       |-- 1001_01_MR_SpinEchoFieldMap2_AP.json
|               |   |       |-- 1001_01_MR_SpinEchoFieldMap2_AP.nii.gz
|               |   |       |-- 1001_01_MR_SpinEchoFieldMap2_PA.json
|               |   |       |-- 1001_01_MR_SpinEchoFieldMap2_PA.nii.gz
|               |   |       |-- 1001_01_MR_rfMRI_REST2_PA.json
|               |   |       |-- 1001_01_MR_rfMRI_REST2_PA_InitialFrames.nii.gz
|               |   |       |-- 1001_01_MR_rfMRI_REST2_PA_SBRef.json
|               |   |       |-- 1001_01_MR_rfMRI_REST2_PA_SBRef.nii.gz
|               |   |       `-- OTHER_FILES
|               |   |           |-- 1001_01_MR_BIAS_32CH.json
|               |   |           |-- 1001_01_MR_BIAS_32CH.nii.gz
|               |   |           |-- 1001_01_MR_BIAS_BC.json
|               |   |           `-- 1001_01_MR_BIAS_BC.nii.gz
|               |   |-- nii
|               |   |   |-- 01.nii.gz
|               |   |   |-- 02.nii.gz
|               |   |   |-- 03.nii.gz
|               |   |   |-- 04.nii.gz
|               |   |   |-- 05.nii.gz
|               |   |   |-- 06.nii.gz
|               |   |   |-- 07.nii.gz
|               |   |   |-- 08.nii.gz
|               |   |   |-- 09.nii.gz
|               |   |   |-- 10.nii.gz
|               |   |   `-- 11.nii.gz
|               |   |-- session.txt
|               |   `-- session_hcp.txt
|               |-- QC
|               |-- archive
|               |   |-- BIDS
|               |   |-- EEG
|               |   |-- HCPLS
|               |   |-- MR
|               |   `-- behavior
|               |-- inbox
|               |   |-- BIDS
|               |   |-- EEG
|               |   |-- HCPLS
|               |   |-- MR
|               |   |-- behavior
|               |   |-- concs
|               |   `-- events
|               `-- specs
|                   |-- hcp_mapping.txt
|                   `-- parameters_example.txt
|-- qunex_container
|-- qunexcontainer
|   |-- README.md
|   `-- qunex_suite-0.100.0.sif
`-- runlogs
    |-- Log-import_hcp_2024-08-12_15.22.32.750548.log
    |-- Log-import_hcp_2024-08-12_15.22.52.636098.log
    `-- Log-import_hcp_2024-08-12_15.24.14.945316.log ```


Thanks,
Mona

Hi,

This will be hard to resolve without knowing what exactly is in the 1001_01_MR.zip file. Is there a way for you to maybe share this image archive? Alternatively, you could list the contents here.

Best, Jure

Hi,

The contents of the folder 1001_01_MR are what I mentioned before:

rfMRI_REST1_AP
rfMRI_REST1_PA
rfMRI_REST2_AP
rfMRI_REST2_PA
T1w_MPR
T2w_SPC

Thanks,
Mona

Hi Mona,

From your import, you can see that QuNex is not correctly identifying the session and the subject. HCP has a specific naming convention for unprocessed data and QuNex is preset for that. The list you provided is not sufficient for me to resolve this. I would need the whole folder structure. I also do not know if all is well with the archive you are importing. Could you maybe share the full archive or provide the full printout of the contents (the tree command on the folder in linux/mac os should to the trick).

In import_hcp the --nameformat is used for fine tuning the import structure. But it uses regular expressions and might be thus a bit technical.

        --nameformat (str, default '(?P<subject_id>[^/]+?)_(?P<session_name>[^/]+?)/unprocessed/(?P<data>.*)'):
            An optional parameter that contains a regular expression pattern
            with named fields used to extract the subject and session
            information based on the file paths and names. The pattern has to
            return the groups named:

            - 'subject_id'   ... the id of the subject
            - 'session_name' ... the name of the session
            - 'data'         ... the rest of the path with the sequence related
              files.

Best, Jure

Hi Jure,

Here is the folder structure for this particular subject:

file:///sc/arion/projects/WenglerLab_SczEnsembles/data/HCP_EP/Package_1231451/imagingcollection01/1001_01_MR
`-- unprocessed
    |-- T1w_MPR
    |   |-- 1001_01_MR_T1w_MPR.json
    |   |-- 1001_01_MR_T1w_MPR.nii.gz
    |   `-- OTHER_FILES
    |       |-- 1001_01_MR_BIAS_32CH.json
    |       |-- 1001_01_MR_BIAS_32CH.nii.gz
    |       |-- 1001_01_MR_BIAS_BC.json
    |       |-- 1001_01_MR_BIAS_BC.nii.gz
    |       |-- 1001_01_MR_SpinEchoFieldMap1_AP.json
    |       |-- 1001_01_MR_SpinEchoFieldMap1_AP.nii.gz
    |       |-- 1001_01_MR_SpinEchoFieldMap1_PA.json
    |       |-- 1001_01_MR_SpinEchoFieldMap1_PA.nii.gz
    |       |-- 1001_01_MR_T1w_MPR_Norm.json
    |       |-- 1001_01_MR_T1w_MPR_Norm.nii.gz
    |       `-- session_report.csv
    |-- T2w_SPC
    |   |-- 1001_01_MR_T2w_SPC.json
    |   |-- 1001_01_MR_T2w_SPC.nii.gz
    |   `-- OTHER_FILES
    |       |-- 1001_01_MR_BIAS_32CH.json
    |       |-- 1001_01_MR_BIAS_32CH.nii.gz
    |       |-- 1001_01_MR_BIAS_BC.json
    |       |-- 1001_01_MR_BIAS_BC.nii.gz
    |       |-- 1001_01_MR_SpinEchoFieldMap1_AP.json
    |       |-- 1001_01_MR_SpinEchoFieldMap1_AP.nii.gz
    |       |-- 1001_01_MR_SpinEchoFieldMap1_PA.json
    |       |-- 1001_01_MR_SpinEchoFieldMap1_PA.nii.gz
    |       |-- 1001_01_MR_T2w_SPC_Norm.json
    |       `-- 1001_01_MR_T2w_SPC_Norm.nii.gz
    |-- rfMRI_REST1_AP
    |   |-- 1001_01_MR_SpinEchoFieldMap1_AP.json
    |   |-- 1001_01_MR_SpinEchoFieldMap1_AP.nii.gz
    |   |-- 1001_01_MR_SpinEchoFieldMap1_PA.json
    |   |-- 1001_01_MR_SpinEchoFieldMap1_PA.nii.gz
    |   |-- 1001_01_MR_rfMRI_REST1_AP.json
    |   |-- 1001_01_MR_rfMRI_REST1_AP.nii.gz
    |   |-- 1001_01_MR_rfMRI_REST1_AP_InitialFrames.nii.gz
    |   |-- 1001_01_MR_rfMRI_REST1_AP_SBRef.json
    |   |-- 1001_01_MR_rfMRI_REST1_AP_SBRef.nii.gz
    |   `-- OTHER_FILES
    |       |-- 1001_01_MR_BIAS_32CH.json
    |       |-- 1001_01_MR_BIAS_32CH.nii.gz
    |       |-- 1001_01_MR_BIAS_BC.json
    |       `-- 1001_01_MR_BIAS_BC.nii.gz
    |-- rfMRI_REST1_PA
    |   |-- 1001_01_MR_SpinEchoFieldMap1_AP.json
    |   |-- 1001_01_MR_SpinEchoFieldMap1_AP.nii.gz
    |   |-- 1001_01_MR_SpinEchoFieldMap1_PA.json
    |   |-- 1001_01_MR_SpinEchoFieldMap1_PA.nii.gz
    |   |-- 1001_01_MR_rfMRI_REST1_PA.json
    |   |-- 1001_01_MR_rfMRI_REST1_PA.nii.gz.partial
    |   |-- 1001_01_MR_rfMRI_REST1_PA_InitialFrames.nii.gz
    |   |-- 1001_01_MR_rfMRI_REST1_PA_SBRef.json
    |   |-- 1001_01_MR_rfMRI_REST1_PA_SBRef.nii.gz
    |   `-- OTHER_FILES
    |       |-- 1001_01_MR_BIAS_32CH.json
    |       |-- 1001_01_MR_BIAS_32CH.nii.gz
    |       |-- 1001_01_MR_BIAS_BC.json
    |       `-- 1001_01_MR_BIAS_BC.nii.gz
    |-- rfMRI_REST2_AP
    |   |-- 1001_01_MR_SpinEchoFieldMap2_AP.json
    |   |-- 1001_01_MR_SpinEchoFieldMap2_AP.nii.gz
    |   |-- 1001_01_MR_SpinEchoFieldMap2_PA.json
    |   |-- 1001_01_MR_SpinEchoFieldMap2_PA.nii.gz
    |   |-- 1001_01_MR_rfMRI_REST2_AP.json
    |   |-- 1001_01_MR_rfMRI_REST2_AP.nii.gz.partial
    |   |-- 1001_01_MR_rfMRI_REST2_AP_InitialFrames.nii.gz
    |   |-- 1001_01_MR_rfMRI_REST2_AP_SBRef.json
    |   |-- 1001_01_MR_rfMRI_REST2_AP_SBRef.nii.gz
    |   `-- OTHER_FILES
    |       |-- 1001_01_MR_BIAS_32CH.json
    |       |-- 1001_01_MR_BIAS_32CH.nii.gz
    |       |-- 1001_01_MR_BIAS_BC.json
    |       `-- 1001_01_MR_BIAS_BC.nii.gz
    `-- rfMRI_REST2_PA
        |-- 1001_01_MR_SpinEchoFieldMap2_AP.json
        |-- 1001_01_MR_SpinEchoFieldMap2_AP.nii.gz
        |-- 1001_01_MR_SpinEchoFieldMap2_PA.json
        |-- 1001_01_MR_SpinEchoFieldMap2_PA.nii.gz
        |-- 1001_01_MR_rfMRI_REST2_PA.json
        |-- 1001_01_MR_rfMRI_REST2_PA_InitialFrames.nii.gz
        |-- 1001_01_MR_rfMRI_REST2_PA_SBRef.json
        |-- 1001_01_MR_rfMRI_REST2_PA_SBRef.nii.gz
        `-- OTHER_FILES
            |-- 1001_01_MR_BIAS_32CH.json
            |-- 1001_01_MR_BIAS_32CH.nii.gz
            |-- 1001_01_MR_BIAS_BC.json
            `-- 1001_01_MR_BIAS_BC.nii.gz ```

I'm sending you the folder structure of the archive as a text file since it's huge.

Thanks,
Mona
[data archive tree.txt|attachment](upload://epOsgVVIBlQlKlYAhu8PZPdDgko.txt) (823.1 KB)

You can see that there are some “issues” with the imaging data. For example rfMRI_REST1_AP has 1001_01_MR_rfMRI_REST1_AP.nii.gz which is the actual raw BOLD image. But rfMRI_REST1_PA does not have it. It has something called 1001_01_MR_rfMRI_REST1_PA.nii.gz.partial. This partial part probably depicts that the acquisition was not complete? The partial suffix is not something that is standard and QuNex does not support this, it is looking for .nii.gz files which is the norm. I assume the same holds for other images that do not get imported. What you could do is rename this partial images into .nii.gz, so

1001_01_MR_rfMRI_REST1_PA.nii.gz.partial => 1001_01_MR_rfMRI_REST1_PA.nii.gz
1001_01_MR_rfMRI_REST2_AP.nii.gz.partial => 1001_01_MR_rfMRI_REST2_AP.nii.gz
1001_01_MR_rfMRI_REST2_PA.nii.gz.partial => 1001_01_MR_rfMRI_REST2_PA.nii.gz

You can use the mv or cp commands in UNIX to achieve this.

But you need to be careful and consult the person who prepared this archive about what the partial suffix actually means.

Best, Jure

The .partial extension most likely means that the file has not been fully downloaded. I suggest checking if the files have been downloaded or copied correctly and redoing the download or copying if necessary.

Hello,

I redownloaded the data folder from the archive and now there are no partial files in the folder. I was able to create the study. However, when I tried to run import_hcp, it cannot identify the inbox folder. I was able to successfully run it for another subject previously.

This is what I did to redownload. First I deleted the incomplete 1001_01_MR which was under the imagingcollection01 folder. Then I redownloaded the 1001_01_MR data, it got downloaded as a package under imagingcollection01 . I simply moved the new 1001_01_MR data folder to the imagingcollection01, followed by deleted the package. The ‘tree’ looks just like before, only with complete files. Then I ran import_hcp and got the error.

Call:


qunex import_hcp --sessionsfolder="/sc/arion/projects/WenglerLab_SczEnsembles//qunex/quickstart/HCP_EP/1001_01/sessions" --inbox="/sc/arion/projects/WenglerLab_SczEnsembles//data/HCP_EP/Package_1231451/imagingcollection01" --archive="leave" --sessions="1001_01_MR" ```

Logs:

``` started running import_hcp at 2024-08-19 22:26:11, track progress in /sc/arion/projects/WenglerLab_SczEnsembles/qunex/quickstart/HCP_EP/1001_01/processing/logs/comlogs/tmp_import_hcp_2024-08-19_22.26.11.694845.log
call: gmri import_hcp sessionsfolder="/sc/arion/projects/WenglerLab_SczEnsembles//qunex/quickstart/HCP_EP/1001_01/sessions" inbox="/sc/arion/projects/WenglerLab_SczEnsembles//data/HCP_EP/Package_1231451/imagingcollection01" archive="leave" sessions="1001_01_MR"
-----------------------------------------
Running import_hcp
==================
---> identifying files in /sc/arion/projects/WenglerLab_SczEnsembles//data/HCP_EP/Package_1231451/imagingcollection01

---> ERROR in completing import_hcp:
     Inbox does not exist
     The specified inbox [/sc/arion/projects/WenglerLab_SczEnsembles//data/HCP_EP/Package_1231451/imagingcollection01] does not exist!
     Please check your path! ```

Thanks,
Mona

Hi,

  1. Before running import_hcp you ran
qunex_container \
    --studyfolder="/sc/arion/projects/WenglerLab_SczEnsembles/qunex/quickstart/HCP_EP/1001_01" \
    --container="${QUNEX_CONTAINER}"

right?

  1. Maybe QuNex container does not have access to the folder. To assure it has it, you can add the --bind parameter to the call:
qunex_container import_hcp \
    --sessionsfolder="/sc/arion/projects/WenglerLab_SczEnsembles/qunex/quickstart/HCP_EP/1001_01/sessions" \
    --inbox="/sc/arion/projectsWenglerLab_SczEnsembles/data/HCP_EP/Package_1231451/imagingcollection01" \
    --archive="leave" \
    --sessions="1001_01_MR" \
    --bind="/sc/arion/projectsWenglerLab_SczEnsembles:/sc/arion/projectsWenglerLab_SczEnsembles" \
    --container="${QUNEX_CONTAINER}"
  1. Previously the --inbox settings was --inbox="/Users/monamunsi/softwares/qunex/data/HCP_EP" so something must have changed there … Can you paste the folder structure of your data folder (or HCP_EP folder) here, you can get only folders by using tree -d.

Best, Jure

Hi Jure,

Yes I ran create_study before running import_hcp. I tried using ‘bind’ but it still cannot identify the inbox. The error looks exactly how it was in the previous message. The inbox setting has not changed. All the $RAW_DATA are still at: /sc/arion/projectsWenglerLab_SczEnsembles/data/HCP_EP/Package_1231451/imagingcollection01.

The tree is in the attached file. The tree
tree.txt (155.5 KB)
has been created at: /sc/arion/projects/WenglerLab_SczEnsembles

Thanks,
Mona

This seems like an issue with permissions or access on your end. QuNex is pretty clear that from his perspective the /sc/arion/projects/WenglerLab_SczEnsembles/data/HCP_EP/Package_1231451/imagingcollection01 does not exist.

What I noticed from your tree structure is that the import for 1002_01 (1002_01_MR) looks OK? It has all bolds and everything.

If you let me know which container you are using (Docker/Singularity), I can write down the instructions for you to manually enter it and inspect what is happening.

Best, Jure

Hi Jure,

Yes it worked for 1002_01 without any issues. I am using Singularity container: 0.100.0.

Thanks,
Mona

Hi,

It is super weird that it worked with 1002_01 as on paper the structure and the location are exactly the same. Here is how you can investigate this:

# enter the QuNex container
singularity run --cleanenv qunex_suite-0.100.0.sif bash

# go to the data root folder
cd /sc/arion/projects/WenglerLab_SczEnsembles/data/HCP_EP/Package_1231451/imagingcollection01

# if you are unable to get into that folder then the container does not have access to it
# you can try going to a higher level folder then and investigate there
# e.g. cd /sc/arion/projects/WenglerLab_SczEnsembles/data/HCP_EP

# check permissions
ls -l

# go to the study folder
cd 1001_01_MR

# check permissions
ls -l

Jure

Hi,

For imagingcollection01, the permissions are:

drwxrws--- 3 munsim03 WenglerLab_SczEnsembles  4096 Aug  9 14:14 1002_01_MR
drwxrws--- 3 munsim03 WenglerLab_SczEnsembles  4096 Aug  9 14:14 1003_01_MR
drwxrws--- 3 munsim03 WenglerLab_SczEnsembles  4096 Aug  9 14:15 1004_01_MR
drwxrws--- 3 munsim03 WenglerLab_SczEnsembles  4096 Aug  9 14:17 1005_01_MR
drwxrws--- 3 munsim03 WenglerLab_SczEnsembles  4096 Aug  9 14:15 1006_01_MR
drwxrws--- 3 munsim03 WenglerLab_SczEnsembles  4096 Aug  9 14:14 1007_01_MR
drwxrws--- 3 munsim03 WenglerLab_SczEnsembles  4096 Aug  9 14:15 1009_01_MR ```

As you can see, the permissions for 1001_01_MR looks a little different. But another member in my group could easily run import_hcp without any issues. He did not even had to use the bind parameter. ls -l on 1001_01_MR gave similar result.

On the other hand, I tried running import_hcp on 1003_01_MR, and I got the same error as 1001_01_MR but I could resolve it with bind parameter. Why do I get this error for 1003_01_MR, when I had absolutely no issues with 1002_01_MR?

 Thanks,
Mona

It is hard to say, because these are not QuNex issues but issues on your system … Anything permissions or access related is outside of QuNex’s scope and not something we can handle on our end.

The list of folders you pasted is missing the problematic session, the list starts with 1002_01_MR … So I cannot see what is happening. Please review your posts before submitting as similar issues are present in your previous posts as well and it is much harder for me to review your issues.

To sort out the permissions, you could do this:

cd /sc/arion/projects/WenglerLab_SczEnsembles/data/HCP_EP/Package_1231451/
chmod -R 775 imagingcollection01

After you run this, all folders should have sufficient permissions for QuNex to access.

Best, Jure

Hi,

The above issue got resolved. Thank you for all the help! I have question regarding processing HCP data in small batches, preferably in batches of 10 subjects at a time, due to space constraint. I have 183 subjects in total. Instead of writing which subjects(sessions) to process in the ‘–sessions’, is there a way to loop through 10 subjects at a time? For example, if I have to process 1001_01_MR till 1010_01_MR, I want to run a for loop on the entries of a text file which contains just the names of the sessions. I tried testing it with a bash script looping only on the import_hcp but it gave the following errors. The sif file is in the path and I have processed 4 sessions separately so far. I have written the portion of the bash script in the call section.

#!/bin/bash

# This script reads each entry from the HCP_EP subject list and runs the qunex commands to minimally process dataset.

cd /sc/arion/projects/WenglerLab_SczEnsembles/qunex

#the file to read from
file="HCPepSubjects.txt";

#Total number of subjects (lines) in the dataset
#wc -l HCPepSubjects.txt

line_count=$(wc -l < HCPepSubjects.txt);
echo $line_count

start_line=3
end_line=13

# Initialize a line counter
line_num=0

# Loop through the file and read line by line
while IFS= read -r line
do
    ((line_num++))  # Increment line number

    # Skip lines until start_line is reached
    if (( line_num < start_line )); then
        continue
    fi

    # Stop when we go past end_line
    if (( line_num > end_line )); then
        break
    fi

    #Starting qunex pipeline
    export SESSIONS="$line"
    echo $SESSIONS

    qunex_container import_hcp \
    --sessionsfolder="${STUDY}/sessions" \
    --sessions="$SESSIONS" \
    --inbox="${RAW_DATA}" \
    --processed_data="$WORK_DIR/preprocessed/$SESSIONS"
    --archive="leave" \
    --bind="$WORK_DIR:$WORK_DIR"
    --container="$QUNEX_CONTAINER"

done < "$file"



Logs: 

ERROR: No Singularity/Apptainer image or Docker container name specified through the --container parameter!
./MinProcess_HCPep.sh: line 55: --archive=leave: command not found
./MinProcess_HCPep.sh: line 56: --container=/sc/arion/projects/WenglerLab_SczEnsembles/qunex/qunexcontainer/qunex_suite-0.100.0.sif: No such file or directory ```

Thanks,
Mona