Using a UserAgent Response, Mojo::Messag. like an IO::Handle object

[bschmalhofer]

Hi,
I'm working on an web app that serves files from an S3 like storage. The application itself is based on Plack. The retrieval from S3 is done with Mojo::UserAgent. Currently I'm fetching the files from S3 into memory and then pass the content on to the PSGI response. This means that there is an intermediate step where the complete file is stored in memory, which is not nice. But PSGI supports returning the response body as an IO::Handle like object. So I'm wondering whether I can use to response of the Mojo::UserAgent as a
IO::Handle like object. Something like Mojo::Message->to_fh, analogous to Mojo::Message->to_string. Or is this a bad idea altogether? In order to avoit the memory footprint I could save the file locally and serve it from there.

For example code see RetrieveObject()https://github.com/RotherOSS/otobo/blob/rel-10_1/Kernel/System/Storage/S3.pm.

[kraih]

The proxy helpers stream response content, just take a look how they deal with this.

[bschmalhofer]

Thanks for the tip. With my present expertise this looks bit too low level. I'll settle with making small steps in refactoring. The first step will be checking whether the asset is in memory or in a file. When it is in memory I can serve the full content. If it is a file, I can serve an open file handle. A later step could be a tiny Mojo App that uses proxy.start_p directly. That tiny app would probably be needed to be wrapped by Mojo::Server::PSGI.

[fskale]

The used S3 module doesn't support multipart uploading, and proper encoding (AWS)
Files > 8MB have to split to be AWS compatible…
Perhaps my S3 POC (BACKUP/RESTORE) script i've written years ago may be of help. (Also supports concurrency)
https://gist.github.com/fskale/23fe035b271e73135cb63cc9acae8a9b

[bschmalhofer]

Thanks for the tip. Up to now I did not even know that there was such a thing as multipart uploads. The actually used S3 storage will likely be MinIO Object Storage. I'll need to check what the size restriction are with that product.
Kernel::System::Storage::S3 is not intended as a general S3 module, it just needs to provide the functionality that OTOBO needs. In the concrete cast it expects that outgoing data is encoded but the UTF8 flag might be set. Incoming data mustn't be decoded, this is done outside the module. I suppose that this is not very sensible, but compatiblilty with existing behavior wins in this case.

[Grinnz]

Note that the UTF8 flag does not indicate whether a string is encoded, you must keep track of that yourself. The UTF8 flag only indicates how Perl is storing the string within itself, and anything relying on it separately is a bug. (There are unfortunately such bugs in Perl core)

[bschmalhofer]

Yes, I'm aware the OTOBO is misbehaving in this regard, especially Kernel::System::Encode::EncodeInput() exptlicitly activating the UTF-8 flag without really knowing how the input is actually encoded. And of course the method name EncodeInput() does not make sense at all. It boils down to that input from the web server and from strings in the database are actually UTF-8 encoded, and that the content of attachment is of unknown encoding. But it is (mostly) assured that the internal representation never changes.

[Grinnz]

The larger issue is that the internal representation is not directly related to the logical contents as seen by operations such as eq, regex, and print - the UTF8 bit only indicates how to translate the internal representation to the logical contents. The codepoints of the string can freely represent bytes or characters regardless of which way the bit is flipped, and Perl may change the internal representation of a string arbitrarily, the only restriction being downgraded Perl strings (UTF8 bit off) cannot physically store codepoints above U+FF (but this still allows non-ascii codepoints thus leading to bugs when misinterpreted). But I am not familiar with the modules to offer any advice.