Efficiently handle metadata for large static assets

[peoplemerge]

Big thanks for contributing this project. I haven't set it up yet and I'm already a big fan.

I looked over the lambda logic, and I have a question / a feature request / feedback. Suppose we want to create a site for a photographer. There is some text content, but largely organized images. It looks like the lambda logic copies in the entire s3 input bucket with content. That doesn't seem to fit so nicely if you have gigs of content.

Is this a scenario you've already considered? I'm reading about hugo handling static content (I'm new to hugo), so I'm trying to out myself, but input would be appreciated.

[peoplemerge]

Also on a related question, suppose you are uploading a vast amount of small assets to s3 simultaneously in a multithreaded manner. Won't the lambda function run every time an upload occurs, regardless of whether another lambda instance is running simultaneously? I've been wondering if there was a mechanism to handle this sort of concurrency, but haven't found anything in lambda itself. Mostly curious as a thought exercise, but it seems like lots of lambda might follow the general case of "do something on receipt of the last message, skipping any in between."

[ryansb]

So I'll answer your first question first:

Right now, if your item is in a static directory (where you would likely store images) it would only copy the image to the proper output path, so you should be fine creating a photographer site since static and hugo site content are handled differently.

For your second question:

Lots of small concurrent uploads are a problem, and I actually handle it by disabling the event source when I upload a big batch of files (e.g. the first upload of a site) and then uploading a final file to trigger the rebuild of the site. I'm not sure this is an option, but if there were time-based batching options, that would work.