I have followed the quickstart documentation to pull the docker container image (0.100.0) to my Mac M3 pc. I ran the create_study command successfully. Although I got the warning that the requested image’s platform(linux/amd64) does not match the host platform(linux/arm64/v8). I checked that Rosetta is installed in my machine. Can this mismatch create any problem later on? If yes, what is the workaround? Next when I ran the command import_hcp, I got an error that the extra argument masterinbox is invalid. I have, for the time being, downloaded one subject’s unprocessed data from NIMH data archive. I have tried running qunex import_hcp --h but it’s not identifying the command.
Below I have given the main portions from the logs. Please advice.
WARNING: The requested image’s platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Usually the mismatch results only in the warning and it works fine later on. We are not compiling the container for the newer Mac architecture specifically because the container is mainly used on compute servers and not so much on laptops. Keep me posted if there are any issues with this down the line.
To get help, you also need to use the qunex_container call. So, to get help for import_hcp you can use:
Note that you can probably omit the python3 at the beginning as usually python3 is the default one. All help is also available at Background — QuNex documentation so you can get the same info in an easier-to-read format there.
For import_hcp this is import_hcp — QuNex documentation. There is a bit of an inconsistency here on our end as the import_dicom command has a parameter called --masterinbox while import_hcp uses --inbox. Try swapping that.
Also, when making posts, you can encapsulate code blocks and log excerpts with triple back ticks (repeat ` 3 times to start the block and the same to end it), this way it is much easier to read. For example, we get:
Thank you for taking the time to respond to my questions.
When I tried running qunex_container script without python3, it gave the error that zsh could not find qunex_container. But it ran when I added python3. Not sure what’s going on.
I made the change from masterinbox to inbox and reran the import_hcp. It threw the error that no files were found in the data folder. Below are the calls and logs.
I am also attaching the folder structure for your reference.
Your post is truncated a bit on the top so I am doing some guesswork here. Based on the folder structure I think the settings for import_hcp should be:
The import_hcp worked after I made the suggested changes. However, when I tried to run create_session_info, I was getting errors for the mapping file. A little bit of digging around showed that the session.txt file thus generated doesn’t have all the rest bolds. The unprocessed dataset has the following files: rfMRI_REST1_AP rfMRI_REST1_PA rfMRI_REST2_AP rfMRI_REST2_PA T1w_MPR T2w_SPC.
When I run import_hcp, it’s not mapping all the nifti files. I’m going to put everything in the Calls portion. I am pasting the session.txt file as well. Basically, import_hcp is skipping a lot of files and I would appreciate if you could help me fixing it. I am also including the folder structure.
This will be hard to resolve without knowing what exactly is in the 1001_01_MR.zip file. Is there a way for you to maybe share this image archive? Alternatively, you could list the contents here.
From your import, you can see that QuNex is not correctly identifying the session and the subject. HCP has a specific naming convention for unprocessed data and QuNex is preset for that. The list you provided is not sufficient for me to resolve this. I would need the whole folder structure. I also do not know if all is well with the archive you are importing. Could you maybe share the full archive or provide the full printout of the contents (the tree command on the folder in linux/mac os should to the trick).
In import_hcp the --nameformat is used for fine tuning the import structure. But it uses regular expressions and might be thus a bit technical.
--nameformat (str, default '(?P<subject_id>[^/]+?)_(?P<session_name>[^/]+?)/unprocessed/(?P<data>.*)'):
An optional parameter that contains a regular expression pattern
with named fields used to extract the subject and session
information based on the file paths and names. The pattern has to
return the groups named:
- 'subject_id' ... the id of the subject
- 'session_name' ... the name of the session
- 'data' ... the rest of the path with the sequence related
files.
You can see that there are some “issues” with the imaging data. For example rfMRI_REST1_AP has 1001_01_MR_rfMRI_REST1_AP.nii.gz which is the actual raw BOLD image. But rfMRI_REST1_PA does not have it. It has something called 1001_01_MR_rfMRI_REST1_PA.nii.gz.partial. This partial part probably depicts that the acquisition was not complete? The partial suffix is not something that is standard and QuNex does not support this, it is looking for .nii.gz files which is the norm. I assume the same holds for other images that do not get imported. What you could do is rename this partial images into .nii.gz, so
The .partial extension most likely means that the file has not been fully downloaded. I suggest checking if the files have been downloaded or copied correctly and redoing the download or copying if necessary.
I redownloaded the data folder from the archive and now there are no partial files in the folder. I was able to create the study. However, when I tried to run import_hcp, it cannot identify the inbox folder. I was able to successfully run it for another subject previously.
This is what I did to redownload. First I deleted the incomplete 1001_01_MR which was under the imagingcollection01 folder. Then I redownloaded the 1001_01_MR data, it got downloaded as a package under imagingcollection01 . I simply moved the new 1001_01_MR data folder to the imagingcollection01, followed by deleted the package. The ‘tree’ looks just like before, only with complete files. Then I ran import_hcp and got the error.
Call:
qunex import_hcp --sessionsfolder="/sc/arion/projects/WenglerLab_SczEnsembles//qunex/quickstart/HCP_EP/1001_01/sessions" --inbox="/sc/arion/projects/WenglerLab_SczEnsembles//data/HCP_EP/Package_1231451/imagingcollection01" --archive="leave" --sessions="1001_01_MR" ```
Logs:
``` started running import_hcp at 2024-08-19 22:26:11, track progress in /sc/arion/projects/WenglerLab_SczEnsembles/qunex/quickstart/HCP_EP/1001_01/processing/logs/comlogs/tmp_import_hcp_2024-08-19_22.26.11.694845.log
call: gmri import_hcp sessionsfolder="/sc/arion/projects/WenglerLab_SczEnsembles//qunex/quickstart/HCP_EP/1001_01/sessions" inbox="/sc/arion/projects/WenglerLab_SczEnsembles//data/HCP_EP/Package_1231451/imagingcollection01" archive="leave" sessions="1001_01_MR"
-----------------------------------------
Running import_hcp
==================
---> identifying files in /sc/arion/projects/WenglerLab_SczEnsembles//data/HCP_EP/Package_1231451/imagingcollection01
---> ERROR in completing import_hcp:
Inbox does not exist
The specified inbox [/sc/arion/projects/WenglerLab_SczEnsembles//data/HCP_EP/Package_1231451/imagingcollection01] does not exist!
Please check your path! ```
Thanks,
Mona
Previously the --inbox settings was --inbox="/Users/monamunsi/softwares/qunex/data/HCP_EP" so something must have changed there … Can you paste the folder structure of your data folder (or HCP_EP folder) here, you can get only folders by using tree -d.
Yes I ran create_study before running import_hcp. I tried using ‘bind’ but it still cannot identify the inbox. The error looks exactly how it was in the previous message. The inbox setting has not changed. All the $RAW_DATA are still at: /sc/arion/projectsWenglerLab_SczEnsembles/data/HCP_EP/Package_1231451/imagingcollection01.
The tree is in the attached file. The tree tree.txt (155.5 KB)
has been created at: /sc/arion/projects/WenglerLab_SczEnsembles
This seems like an issue with permissions or access on your end. QuNex is pretty clear that from his perspective the /sc/arion/projects/WenglerLab_SczEnsembles/data/HCP_EP/Package_1231451/imagingcollection01 does not exist.
What I noticed from your tree structure is that the import for 1002_01 (1002_01_MR) looks OK? It has all bolds and everything.
If you let me know which container you are using (Docker/Singularity), I can write down the instructions for you to manually enter it and inspect what is happening.
It is super weird that it worked with 1002_01 as on paper the structure and the location are exactly the same. Here is how you can investigate this:
# enter the QuNex container
singularity run --cleanenv qunex_suite-0.100.0.sif bash
# go to the data root folder
cd /sc/arion/projects/WenglerLab_SczEnsembles/data/HCP_EP/Package_1231451/imagingcollection01
# if you are unable to get into that folder then the container does not have access to it
# you can try going to a higher level folder then and investigate there
# e.g. cd /sc/arion/projects/WenglerLab_SczEnsembles/data/HCP_EP
# check permissions
ls -l
# go to the study folder
cd 1001_01_MR
# check permissions
ls -l
drwxrws--- 3 munsim03 WenglerLab_SczEnsembles 4096 Aug 9 14:14 1002_01_MR
drwxrws--- 3 munsim03 WenglerLab_SczEnsembles 4096 Aug 9 14:14 1003_01_MR
drwxrws--- 3 munsim03 WenglerLab_SczEnsembles 4096 Aug 9 14:15 1004_01_MR
drwxrws--- 3 munsim03 WenglerLab_SczEnsembles 4096 Aug 9 14:17 1005_01_MR
drwxrws--- 3 munsim03 WenglerLab_SczEnsembles 4096 Aug 9 14:15 1006_01_MR
drwxrws--- 3 munsim03 WenglerLab_SczEnsembles 4096 Aug 9 14:14 1007_01_MR
drwxrws--- 3 munsim03 WenglerLab_SczEnsembles 4096 Aug 9 14:15 1009_01_MR ```
As you can see, the permissions for 1001_01_MR looks a little different. But another member in my group could easily run import_hcp without any issues. He did not even had to use the bind parameter. ls -l on 1001_01_MR gave similar result.
On the other hand, I tried running import_hcp on 1003_01_MR, and I got the same error as 1001_01_MR but I could resolve it with bind parameter. Why do I get this error for 1003_01_MR, when I had absolutely no issues with 1002_01_MR?
Thanks,
Mona
It is hard to say, because these are not QuNex issues but issues on your system … Anything permissions or access related is outside of QuNex’s scope and not something we can handle on our end.
The list of folders you pasted is missing the problematic session, the list starts with 1002_01_MR … So I cannot see what is happening. Please review your posts before submitting as similar issues are present in your previous posts as well and it is much harder for me to review your issues.
To sort out the permissions, you could do this:
cd /sc/arion/projects/WenglerLab_SczEnsembles/data/HCP_EP/Package_1231451/
chmod -R 775 imagingcollection01
After you run this, all folders should have sufficient permissions for QuNex to access.
The above issue got resolved. Thank you for all the help! I have question regarding processing HCP data in small batches, preferably in batches of 10 subjects at a time, due to space constraint. I have 183 subjects in total. Instead of writing which subjects(sessions) to process in the ‘–sessions’, is there a way to loop through 10 subjects at a time? For example, if I have to process 1001_01_MR till 1010_01_MR, I want to run a for loop on the entries of a text file which contains just the names of the sessions. I tried testing it with a bash script looping only on the import_hcp but it gave the following errors. The sif file is in the path and I have processed 4 sessions separately so far. I have written the portion of the bash script in the call section.
#!/bin/bash
# This script reads each entry from the HCP_EP subject list and runs the qunex commands to minimally process dataset.
cd /sc/arion/projects/WenglerLab_SczEnsembles/qunex
#the file to read from
file="HCPepSubjects.txt";
#Total number of subjects (lines) in the dataset
#wc -l HCPepSubjects.txt
line_count=$(wc -l < HCPepSubjects.txt);
echo $line_count
start_line=3
end_line=13
# Initialize a line counter
line_num=0
# Loop through the file and read line by line
while IFS= read -r line
do
((line_num++)) # Increment line number
# Skip lines until start_line is reached
if (( line_num < start_line )); then
continue
fi
# Stop when we go past end_line
if (( line_num > end_line )); then
break
fi
#Starting qunex pipeline
export SESSIONS="$line"
echo $SESSIONS
qunex_container import_hcp \
--sessionsfolder="${STUDY}/sessions" \
--sessions="$SESSIONS" \
--inbox="${RAW_DATA}" \
--processed_data="$WORK_DIR/preprocessed/$SESSIONS"
--archive="leave" \
--bind="$WORK_DIR:$WORK_DIR"
--container="$QUNEX_CONTAINER"
done < "$file"
Logs:
ERROR: No Singularity/Apptainer image or Docker container name specified through the --container parameter!
./MinProcess_HCPep.sh: line 55: --archive=leave: command not found
./MinProcess_HCPep.sh: line 56: --container=/sc/arion/projects/WenglerLab_SczEnsembles/qunex/qunexcontainer/qunex_suite-0.100.0.sif: No such file or directory ```
Thanks,
Mona