Persistent Data Storage Between Processes

[opalmer]

As #57 mentions, we need a way to persist data between processes. Ideally we want something we can append to so we're not dumping the entire file every time. The specific goal of this issue is to make it possible to facilitate storage of the above.

Some things we should consider or already intend on storing:

• Important events such as:
  • Error messages
  • Shutdowns
  • Restarts
  • Agent Upgrades
• Resource, such as ram and cpu, usage history
• Results or changes made by scheduled events
• Any other kind of data where logging does not provide granular enough information or where we want data to be available for debugging and metrics purposes.

[guidow]

I think for just making sure tasks don't end up in limbo because of an agent restart, it's sufficient if the agent includes its current list of tasks in every announce and reannounce. The master can then easily catch a discrepancy between that list and what it has in its database.
No persistent storage on the agent side is needed for that.

[opalmer]

Yeah you're probably right that it will be sufficient enough, I've remove that point from this issue. I'm also moving this issue off of 0.8.0 because this is an enhancement that we don't yet have an immediate need for.