Table of Contents
This plugin application is used to recursively copy data from a remote host into an analysis root node.
pull_scp is a ChRIS fs-type application that produces data for an analysis tree by copying data off a remote host using (recursive) scp.
Of course this assumes that the user executing this plugin has the correct login credentials to access the resource. Credentials are defined in either:
- a hard-coded .env file in the repo/container
- setting appropriate environment variables
- using plugin CLI arguments
Other than login credentials, this plugin also needs a filepath in the remote user space. All files and directories rooted in this file filepath are copied into this plugin's outputdir.
This plugin/app is not considered or purported to be secure! One deployment vector has the login credentials contained within an .env file in cleartext and copied into the container. Deploying such a container will expose login credentials! For better security, supply login credentials from the CLI or from within environment variables:
export ENVIRONMENT=development
export SSH_REMOTE_HOST=1.1.1.1
export SSH_USERNAME=yourname
export SSH_PASSWORD=yourpassword
export SSH_KEY_FILEPATH=/usr/local/src/key.pub
export SCP_DESTINATION_FOLDER=/tmpMost of the innards of this plugin are lightly adapted from the most excellent paramiko tutorial of Todd Birchard:
docker run --rm fnndsc/pl-pull_scp pull_scp \
[-h] [--help] \
[--json] \
[--man] \
[--meta] \
[--savejson <DIR>] \
[-v <level>] [--verbosity <level>] \
[--version] \
[--username <username>] \
[--password <password>] \
[--host <hostname>] \
[--sshPubKeyFile <pubKeyFile>] \
--filepath <filepath> \
<outputDir>
--filepath <filepath> The path in the <hostname>'s filesystem to pull. This is technically the only required argument of this plugin. All user specific credentials are assumed to be container in the .env file or set in environment variables. Note of course that all the credentials can be overriden with CLI flags. [--username <username>] The username in the remote host. [--password <password>] The <username>'s password to connect to the remote host. [--sshPubKeyFile <pubKeyFile>] The ssh public key file to use in this session. [--host <hostname>] The hostname to access. [-h] [--help] If specified, show help message and exit. [--json] If specified, show json representation of app and exit. [--man] If specified, print (this) man page and exit. [--meta] If specified, print plugin meta data and exit. [--savejson <DIR>] If specified, save json representation file to DIR and exit. [-v <level>] [--verbosity <level>] Verbosity level for app. Not used currently. [--version] If specified, print version number and exit.
Installation is either via docker (recommended) or directly from PyPI (less recommended).
docker pull fnndsc/pl-pull_scpGetting inline help is:
docker run --rm fnndsc/pl-pull_scp pull_scp --manpip install pull_scpFor this use case, copy your public key to /tmp/key.pub and create an environment file /tmp/.env that either contains your login data/credentials or contains "dummy" data. The .env _must_ exist for the app to work -- regardless of the validity of its data.
Since docker is the recommended usagage deployment, the following instructions are docker based.
Being an _FS_ plugin, you should specify an output directory using the -v flag to docker run.
docker run --rm -u $(id -u) \
-v $(pwd)/out:/outgoing \
fnndsc/pl-pull_scp pull_scp \
--username johnnyapple \
--password 'mysecret' \
--host computer.org \
--sshPubKeyFile ~/.ssh/rsa_pub.key \
--filepath /home/johnnyapple/data \
/outgoingBuild the Docker container:
docker build -t local/pl-pull_scp .Run unit tests:
docker run --rm local/pl-pull_scp nosetestsAll relevant login credentials are stored in /tmp/.env in the container. Note this method embeds the .env file in the container where it can be potentially expose credentials!
mkdir out && chmod 777 out
docker run --rm -u $(id -u) --name=pl-pull_scp \
-v $PWD/out:/outgoing \
-it \
local/pl-pull_scp pull_scp -v 1 \
--filepath /home/rudolphpienaar/Desktop \
outgoing
10-15-2021 16:25:38 | INFO: /tmp/key.pub uploaded to 192.168.1.216
_ _
| | |
_ __ _ _| | | ___ ___ _ __
| '_ \| | | | | | / __|/ __| '_ \
| |_) | |_| | | | \__ \ (__| |_) |
| .__/ \__,_|_|_| |___/\___| .__/
| | ______ | |
|_| |______| |_|
Version: X.Y.Z
10-15-2021 16:25:38 | INFO: (remote): du -ksh /home/rudolphpienaar/Desktop: 142M /home/ rudolphpienaar/Desktop
10-15-2021 16:25:39 | INFO: Pulling rudolphpienaar@192.168.1.216:/home/rudolphpienaar/Desktop...
10-15-2021 16:25:53 | INFO: Remote contents pulled to /outgoing
10-15-2021 16:25:53 | INFO: (NOTE: if running in a container, and doing a volume mapping,
the destination dir name might not match the host dirname!)
10-15-2021 16:25:53 | INFO: (local): du -ksh /outgoing: 142M /outgoingCredentials are supplied at run time. Contents of /tmp/.env are not used.
docker run --rm -u $(id -u) --name=pl-pull_scp \
-v $PWD/out:/outgoing \
-it \
local/pl-pull_scp pull_scp -v 1 \
--filepath /home/chris/Pictures \
--host 192.168.1.200 \
--username chris \
--password XXXXXXXXXXX \
/outgoing
10-15-2021 17:05:13 | INFO: /tmp/key.pub uploaded to 192.168.1.200
_ _
| | |
_ __ _ _| | | ___ ___ _ __
| '_ \| | | | | | / __|/ __| '_ \
| |_) | |_| | | | \__ \ (__| |_) |
| .__/ \__,_|_|_| |___/\___| .__/
| | ______ | |
|_| |______| |_|
Version: X.Y.Z
10-15-2021 17:05:14 | INFO: (remote): du -ksh /home/chris/Pictures: 81M /home/chris/Pictures
10-15-2021 17:05:14 | INFO: Pulling chris@192.168.1.200:/home/chris/Pictures...
10-15-2021 17:06:01 | INFO: Remote contents pulled to /outgoing
10-15-2021 17:06:01 | INFO: (NOTE: if running in a container, and doing a volume mapping,
the destination dir name might not match the host dirname!)
10-15-2021 17:06:01 | INFO: (local): du -ksh /outgoing: 156M /outgoing(note that in this example the local directory is larger than the remote. This occurs when the remote directory contains symbolic links -- each symbolic link is actually translated into the target file when pulled)
_-30-_
