Feature request: an API to list only the keys in storage, not their values?

[nvie]

Sometimes it would be useful to only list the "keys" using the Transactional Storage API, and not also having to retreive all of their values. If there are many entries, and they all have a lot of data in their values, then doing a .list() can potentially read N * 128 KiB of unneeded data into memory, which would only be discarded.

For example:

await state.storage.list({ prefix, startAfter, limit, /* etc etc */ });
// new Map([['foo', 'value1'], ['bar', 'value2']])

await state.storage.keys({ prefix, startAfter, limit, /* etc etc */ });
// ['foo', 'bar']

[kentonv]

In the underlying storage on disk, the keys and values are interleaved, so it's not really possible to save any disk I/O by reading just the keys. Perhaps it would allow us to skip some transmission and deserialization but that's not the main cost. So I'm not sure we can really get a meaningful savings here.

With the upcoming sqlite storage engine, you'll of course be able to write a query that only returns one column.

[nvie]

Thanks for getting back to me. Ah, that makes sense. The SQLite storage engine sounds really interesting. Is there any work in progress here that I can follow?

"Perhaps it would allow us to skip some transmission and deserialization but that's not the main cost."

Will this not even make a difference if there are many keys, and the associated values in there are pretty large (close to 128 KiB per key)? We have some storages in production where reading a .list() will hit a memory limit, and so we're forced to read .list() with a limit in a loop just in order to read all the keys (which would fit in memory).