Listen, Poll and Rebuild

The services sit above the filestore level and are designed to be independently updateable, replaceable, duplicable for redundancy and ultimately non-critical to the survival of the stored data and most importantly, non-critical for the tractability of the data the archive holds.

Services that work with the archive are event-driven - the model for their interaction is primarily as Listeners. The search index is kept up-to-date by a listener looking for events showing content added to, removed from or changed within the archive. A message queue of these events is generated by Databank and redistributed according to its configuration, allowing for easy extension and use by other tools. The listener that maintains the Apache Solr index is but one example of this use.

This message queue is network-accessible, as are the APIs to the archive and so curational and/or computational processes that may take minutes or hours to work can be offloaded to remote capacity.

Some services are not stateless - creating a .txt version of a data-packages .pdf files is, but a service that maintains a cache or index certainly isn't. For stateful services, the degradation of interaction is summed up by: 'Listen, Poll or Rebuild'. Primarily, services maintain coherence by listening to events and then updating their state to keep in step with the archived information. Sometimes, this connection is broken or is not trusted, in which case, the files that are affected are polled directly to see if there has been any change. The final level is when there is an unknown amount of conflict between the state of the service and the state of the archive, and so the service can rebuild itself from scratch.

All services that maintain a state MUST be capable of all three models of interaction with the archive.