Guidance for users on what is and is not appropriate for the KVS

[jameshcorbett]

In the past couple of months I've had two users come to me for help with using the KVS to store their job data. One of them mentioned their data would be around ~1Gb per job.

Many years ago when I was working on a workflow manager, when Dong told me that Flux had a KVS, I wanted to use it to store my data. It's probably a common reaction--hear there's a KVS and want to use it to solve all your data problems.

It doesn't seem like we have documentation describing what sorts of use-cases are and are not appropriate for the KVS. I was hoping we could figure that out in this discussion and then I could maybe put some documentation up and link to it from the man pages on the CLI and Python bindings.

[garlick]

That is an excellent suggestion. I had planned to add some KVS docs to the flux-core readthedocs pages but hadn't settled on what that shoud look like. There is an old wiki document that i'd like to upcycle over here but its more for kvs internals.

Some random considerations:

• The default location for a batch job kvs is /tmp which may be a ramdisk with limited space but see the statedir broker attribute.
• By default a batch job's kvs content is not preserved but see flux batch --dump
• We have a place for user defined job data defined in RFC 16
• The KVS back end storage requirements go way up if there is significant "churn" in content, so rewriting a key many times during exection might be considered an antipattern.
• Another antipattern is creating keys with huge values since fetching such a key can cause head of line blocking in the broker.
• flux-archive may be helpful in some use cases, especially where the data becomes input to subsequent jobs because then the stage-in job shell plugin can be used.

Anyway just a few quick thoughts to get things started.