I’m trying to process 10 subjects from HCPYA (HCP1200) as a test. I was able to run the container up till hcp_pre_freesurfer where I got an error that seems to be data linking issue:
---> ERROR: Could not find T1w image file. []
---> ERROR: The requested HCP processing mode is 'HCPStyleData', however, no T2w image was specified!
Consider using LegacyStyleData processing mode.
---> WARNING: No distortion correction method specified.
---> Due to missing files session can not be processed.
Before this create_study, import_hcp, setup_hcp, and create_batch ran sucessfully.
(I had to manually loop through setup_hcp for each session, wasn’t sure if that would be an issue?)
I saw similar errors in this post but I’ve been onboarding HCPYA from scratch using /unpreprocessed. What might be the problem? Is there any configuration that needs to be set manually?
Welcome to the QuNex forum. Sorry for the late reply, I was out of office. At a glance all looks good.
If you manually check the contents, are the files there. For example, do you have
.../sessions/101107_3T/hcpls/T1w_MPR1
And images inside:
.../sessions/100206_3T/nii
I recently processed the 100206_3T session myself. I am pasting the chain of commands below so we can see if something sticks out. I propose that you first try getting 1 session up and running with hcp_pre_freesurfer and then start at scale once we resolve that.
STUDY_FOLDER="/data/jure/qx_study"
qunex_container create_study \
--studyfolder="${STUDY_FOLDER}" \
--bind="/data:/data" \
--container="${QX_CONTAINERS}/qunex_suite-0.100.0.sif"
qunex_container import_hcp \
--sessionsfolder="${STUDY_FOLDER}/sessions" \
--inbox="/data/archive/HCPYA" \
--sessions="100206_3T" \
--archive="leave" \
--nameformat="(?P<subject_id>[^/]+?)/unprocessed/(?P<session_name>.*?)/(?P<data>.*)" \
--hcplsname="hcpya" \
--bind="/data:/data" \
--container="${QX_CONTAINERS}/qunex_suite-0.100.0.sif"
# here you can run create batch before setup_hcp to the facilitate batch for setup_hcp over all sessions
qunex_container create_batch \
--sessionsfolder="${STUDY_FOLDER}/sessions" \
--paramfile="/data/specs/hcpya_parameters.txt" \
--targetfile="${STUDY_FOLDER}/processing/batch.txt" \
--bind="/data:/data" \
--container="${QX_CONTAINERS}/qunex_suite-0.100.0.sif"
qunex_container setup_hcp \
--sessionsfolder="${STUDY_FOLDER}/sessions" \
--sessions="${STUDY_FOLDER}/processing/batch.txt" \
--bind="/data:/data" \
--container="${QX_CONTAINERS}/qunex_suite-0.100.0.sif"
qunex_container hcp_pre_freesurfer \
--sessionsfolder="${STUDY_FOLDER}/sessions" \
--sessions="${STUDY_FOLDER}/processing/batch.txt" \
--bind="/data:/data" \
--container="${QX_CONTAINERS}/qunex_suite-0.100.0.sif"
To run this at scale now, just replacing --sessions="100206_3T" in the import command with a comma separated list of sessions should do the trick. Let me know how this goes.
Thanks for sharing the commands, they’re very helpful! Pre_freesurfer is running now after changing the sessions name.
But there were some parameters I had to work around with - such as manually listing the sessions in setup_hcp and hcp_pre_freesurfer. If I use --session=<path to batch file> as in your command it throws me an error "No sessions specified to process. Please check your batch file, filtering options or sessionids parameter!".
Also I had to add these parameters to hcp_pre_freesurfer: "--hcp_unwarpdir="x" --hcp_seechospacing="0.00058" --hcp_t1samplespacing="0.0000074" --hcp_t2samplespacing="0.0000021" --hcp_seunwarpdir="x" ". I have these values in my parameters.txt but it doesn’t seem like they were detected. Is this normal?
Thank you so much for your time
Selina
List of commands
setenv STUDY_FOLDER "/export/home/selinali/Documents/HCPpipelines_workdir"
setenv QX_CONTAINERS "/export/home/selinali/Documents/qunex"
setenv rootdir "/export/home/selinali/Documents"
setenv PATH "${PATH}:${QX_CONTAINERS}"
rm -rf ${rootdir}/HPpipelines_workdir/
qunex_container create_study \
--studyfolder="${STUDY_FOLDER}" \
--bind="${rootdir}:${rootdir}" \
--container="${QX_CONTAINERS}/qunex_suite-0.100.0.sif"
qunex_container import_hcp \
--sessionsfolder="${STUDY_FOLDER}/sessions" \
--inbox="/export/home/selinali/Documents/study_folder_hcp_" \
--sessions="101107_3T,101107_3T,101309_3T,101915_3T,103111_3T,103414_3T,103818_3T,105014_3T,105115_3T" \
--archive="leave" \
--nameformat="(?P<subject_id>[^/]+?)/unprocessed/(?P<session_name>.*?)/(?P<data>.*)" \
--hcplsname="hcpya" \
--bind="${rootdir}:${rootdir}" \
--container="${QX_CONTAINERS}/qunex_suite-0.100.0.sif"
# here you can run create batch before setup_hcp to the facilitate batch for setup_hcp over all sessions
qunex_container create_batch \
--sessionsfolder="${STUDY_FOLDER}/sessions" \
--paramfile="${QX_CONTAINERS}/data/parameters.txt" \
--targetfile="${STUDY_FOLDER}/processing/batch.txt" \
--bind="${rootdir}:${rootdir}" \
--container="${QX_CONTAINERS}/qunex_suite-0.100.0.sif"
qunex_container setup_hcp \
--sessionsfolder="${STUDY_FOLDER}/sessions" \
--sessions="101107_3T,101107_3T,101309_3T,101915_3T,103111_3T,103414_3T,103818_3T,105014_3T,105115_3T" \
--bind="${rootdir}:${rootdir}" \
--container="${QX_CONTAINERS}/qunex_suite-0.100.0.sif"
qunex_container hcp_pre_freesurfer \
--sessionsfolder="${STUDY_FOLDER}/sessions" \
--batchfile="${STUDY_FOLDER}/processing/batch.txt" \
--sessions="101107_3T,101107_3T,101309_3T,101915_3T,103111_3T,103414_3T,103818_3T,105014_3T,105115_3T" \
--bind="${rootdir}:${rootdir}" \
--hcp_unwarpdir="x" --hcp_seechospacing="0.00058" --hcp_t1samplespacing="0.0000074" --hcp_t2samplespacing="0.0000021" --hcp_seunwarpdir="x" \
--container="${QX_CONTAINERS}/qunex_suite-0.100.0.sif"
Can you upload the batch file so I can take a look at it.
If the parameters are in there, they should be detected and used. Yes, a workaround is to manually provide them, but it should work without this. At least in my case above it works perfectly fine. It seems like something might be wrong in the batch file.
If the line starts with # it means it is not a real setting but just a comment (text). It gets ignored by QuNex. You used the template file without changing it. What is the most clean way is to remove the lines with parameters you do not need and for the parameters you do need, remove the # and the empty space behind it and set the value to what it needs to be. This way, parameter will be properly set from the batch file.
Got it, that makes sense! I’ll try again removing the comments.
I ran through the FreeSurfer stages but had some trouble starting post-freesurfer processing… Seems like it’s not filling in the fmap magnitude and phase file paths.
Actually, I’m a little confused how session.txt and batch.txt are different. Are they doing the same job except one is for single session and the other is for multiple sessions?
The logic is that session.txt is info on the session level, while batch.txt is a concatenation of sessions. The batch files allows us to process many sessions (also in parallel) in a rapid fashion at scale. In the most common case the batch file simply contains all of the sessions. But there are scenarios when I might have multiple session files. For example, my study might have a 100 participants, out of the 100, 90 underwent the full acquisition, while with 10 participants the acquisition was stopped in the middle for some reason. Because of this I might need to treat and process these 10 sessions differently. To accommodate this, I might have 2 batch files, the main one for the 90 sessions and another one for the 10 “outliers”.
There are two session files. The first one is session.txt which gets generated when you import the data. This one includes the data that can be automatically extracted by QuNex from imaging data during the import process. For processing you then often need to inject additional information into these files. This is done with the create_session_info command through the mapping file and the output here is session_hcp.txt. So this file is the session.txt upgraded with additional information required by the HCP pipelines. So I can help you out, please provide the contents of session_hcp.txt for the above session and maybe also the mapping file you used in create_session_info.
