Upload from sftp

[sergejzr]

Hello,

as I mentioned in a mailing list, we are potentially interested in uploading very large files from some labs at our University.

Thanks to @pdurbin for suggesting DVUploader, it looks very promising.

In our case, however the large files are not at the users machine, but on a SFTP directory. I wonder if DVUploader could support such an upload from SFTP directory to a Dataverse.

At the moment I am waiting for a concrete use-case, thus this is not an urgent question for us at the moment. Just wanted to hear about your opinions on this issue.

Thanks
Sergej

[qqmyers]

DVUploader currently only supports upload from the local file system (directly to an S3 store or to the Dataverse server). While adding another mechanism (sftp, rsync, globus) would be possible, I don't know of any plans for that. I also might suggest looking at https://github.com/gdcc/python-dvuploader which might be easier to extend.

(All of these tools just use the Dataverse API which supports two specific options - direct upload via signed S3 URLs provided by Dataverse, and Globus (where Dataverse can monitor a transfer started by an separate tool and add the files to a dataset if/when the transfer succeeds, currently requiring an S3-based Globus endpoint but being extended to use file/tape endpoints) - but it also now has a separate upload-out-of-band option for stores that allows them to be used with any other transfer mechanism. In this case, a separate tool would move the file to the required location in the Dataverse store and then use the direct upload api to add the file(s) to the dataset.)