Skip to content

Latest commit

 

History

History
69 lines (41 loc) · 2.26 KB

README.md

File metadata and controls

69 lines (41 loc) · 2.26 KB
Raw

Automated scan downloading and project organisation

  • download data from HPC to hivemind project folder
  • organise downloaded data for backups
  • copy relevant scans to BIDS format
  • options for ignoring bad runs
  • logs all processes
  • tests for user input, scan ordering

See docs/project_overview.pdf

Iniitialises a project with the structure:

	./PROJECT/data/raw_scans  
	./PROJECT/data/preprocessing  
	./PROJECT/docs/logs

USEAGE:

  1. First prepare your HPC / Hivemind account for passwordless connection and access to the relevant WBIC project (see PREPARATION) below.

  2. Fill out the project_configs.py with the data relevant for your project.

  3. run with python3 run_project.py -download_from_hpc -move_to_preprocessing

    This script will loop through all subjects / sessions specified in project_configs.py and download the data from the HPC (-download_from_hpc), then move the relevant from scans from /raw_scans/ to preprocessing (-move_to_preprocessing).

    If a folder with the zk_id (e..g zk21w7_005) already exists in /raw_scans/ data will not be downloaded. If a matching session already exists in /preprocessing/sub-XXX/ data will not be copied.

  4. If running outside of run_project.py, make sure to init_logging() or logs will not be saved correctly.

PREPARATION:

  1. Setup passworless SSH connection from the hivemind to HPC. The .ssh keys must be stored in your /home directory (this is the default).

    From the hivemind, run:

    ssh-keygen -t rsa cat ~/.ssh/id_rsa.pub | ssh remote_username@server_ip_address "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

    You will be asked to fill in your password during initial setup

  2. Log onto the HPC and check your project access / WBIC. cd to /rds-d5/user/USERNAME/hpc-work/ and make a folder called wbic-data. cd to wbic-data and run:

    module wbic load dcmconv.pl -remoteae PROJECT_ID -date YYYYMMDD -makedir -outtype dicom10 -direct -info

    and try to download a scan. If the download is sucessful, you can delete it from /wbic-data. Do not delete /wbic-data as it will be used as the holding folder for downloading scans from HPC.

connect to ssh make rsa key in home dir make wbic-data and test init logging