web3.storage: don't load large files into memory

[makew0rld]

Currently the upload tool creates CAR files when uploading to web3.storage. This is required to ensure the same CID is used. The library used to create CAR files loads the whole file into memory. This is usually fine, but will come back to bite us when we start ingesting large video files or something.

I have logged an upstream issue for this, although I expect to have to write the fix myself. photon-storage/go-ipfs-car#6

Also, there seems to be some compression in play, so testing future fixes for this should use random data (like from /dev/urandom) rather than a file full of zeros or something.

[makew0rld]

The library used to create CAR files loads the whole file into memory

This is fixed now, creating CAR files doesn't load files into memory when the file is over 1 GiB in size. That seems like a good default so it's what I have for now.

Unfortunately in practice the file still gets loaded into memory because that's what the w3 CLI does during the upload process. I've filed an issue for that: storacha/w3cli#193