Small library to read and upload file contents from a URL to an endpoint simultaneously. The received data chunks are directly sent to the upload endpoint without being saved to storage or memory first, making the process very fast and efficient compared to storing the file before uploading (limited by storage or memory size and the total duration is slower).
Note: while this library might be useful for you, keep in mind that it was developed for learning purposes and requires more testing and improvements.
The library part is the main focus, but a simple main.rs
is also provided for quick command line usage.
use url_file_direct_uploader::{Manager, ManagerUploadOption};
fn main(){
let manager = Manager::new(
"DOWNLOAD_URL",
"UPLOAD_URL",
Some(vec![ // or None to use default values
ManagerUploadOption::FileName(String::from("file.png")),
]),
);
// use the public functions of Manager here to customize
manager.perform(); // or manager.perform_async()
}
cargo run [DOWNLOAD_URL] [UPLOAD_URL] [?UPLOAD_FIELD_NAME] [?UPLOAD_FILE_NAME] [?REQUEST_TYPE]
Example:
cargo run "https://example.com/image.png" "https://example.com/upload" "file" "image.png" "POST"
It might be possible to achieve the same functionality of this library using two piped curl
commands:
curl -s --output - "DOWNLOAD_URL"
to downloadcurl -F file=@- [...]
,curl -T - [...]
or other options to upload
However:
curl -F file=@- [...]
was using a lot of memory as it was storing the data in memory before starting to upload (to determine theContent-Length
for the upload headers)curl -T - [...]
doesn't seem to have the option to specify the file field name, and is possibly not for HTTP uploads- A possible solution is building the request body manually (like this library is doing) instead of relying on
curl
file arguments, and that might work with-T -
or the other options - In the case it's possible (in a similar buffering way), the memory usage was higher and the CPU usage was in close range (tested
curl -T - [...]
despite it not delivering the needed format)
- Cancelling the download request when the upload request couldn't be performed
- Better error reporting and handling
- Documentation