allow to pull data outside dvc project

[link89]

I am on a task on setup a CI job for a tensorflow task that need to start a tensorflow job on a HPC cluster from the CI node. In order to do this I need to send data to the HPC control node first.
In order to reduce uneccessary IO and network usage, I hope that I can just send the *.dvc file to the remote node first, and then run dvc pull command to fetch data to the remote node directly, instead of pulling data on the CI node and scp to the remote node.
Currently I have to run git init and dvc init before I can run dvc pull, it would be greate if dvc could provide a sub command that allow me to do it directly.

[dberenbaum]

Have you taken a look at dvc get as a way to get the data onto the remote node?

[link89]

I found dvc get already support file path instead of git url for edge case, so now what it miss is to allow to change remote url on demand

Notice that the url provided to dvc get above is .. dvc get accepts file system paths as the "URL" to the source repo, for edge cases.

[pared]

@link89 So if you are already using dvc files on the lustre fs, why not checkout the data there and use scp to copy the data itself, not the associated .dvc file to worker node?

[link89]

The dvc files are on my computer and git repo, the actual data is on lustre-fs, and worker node can access the lustre-fs directly. The reason we push data to lustres-fs via ssh instead of scp of orignal files is we are using dvc to share data and manage data's version. There are other developers works on the same project, by doing this way they can easily pull data from lustre-fs to their own machines.

When we submit job to run formal task on worker node, the woker node don't need to pull data via ssh because the it can access lustre-fs directly. But currently there is no clear way to do this.

[pared]

Ok, I think I got it now. One more question:
worker nodes don't have git and dvc installed?

Another thing is whether it is possible to do this currently, and I believe it is, with some coding:

Have you considered using dvc.api?

1. In the control node - use dvc.api.get_url to obtain remote address of required asset - you can provide remote and rev there
2. write a script that uses dvc.api.open to read the assets and write them to particular worker - one case to handle is parsing directory and iteratively obtain and copy included files - get_url will return an address to a json file containing "relpath" relative to directory and its md5 - here you could use relpath to build in-repo path and obtain the address with get_url.

[link89]

Yes, we have git and dvc install. But on HPC we cannot just git clone something as the network is blocked. I have to scp the repo to worker node. Actually we have a work around by adding a remote temporary and use dvc pull to fetch data. But I think it would be clear to have a dedicated command.
I think that's an option to use dvc.api , I will consider this. But I guess it is easier to do this in dvc get command. It's reasonable to have a way to fetch data by providing remote-url and *.dvc files. It's like an enhanced version of dvc get-url that not only fetch data from remote but also reconstruct file structure according to folder that contains *.dvc files.