User cache data format design considerations

User cache data format design considerations

Our cache actually performs two different functions:

1. Bidirectional Document cache: Each cache entry is a document. Each document is associated with metadata such as retrieval time and access time. The cache transfers the document as a whole. While this is similar to the existing HTTP cache, unlike most HTTP caches, our cache only returns data that it has - it does not contact the server to determine liveness. The assumption is that any missing data will be transferred as part of the next background sync. We may need to revisit this assumption if we want more real-time operation. A second variation is that we can cache data intended for saving as well. This is one of the other differences from the HTTP cache. We can, of course, choose to use a HTTP cache for the GET and our code for the PUT. At this time, that seems unnecessarily complex.
2. Sensed data: For this data, what we arguably want is not a cache, but rather, a technique to stream the sensed data to the server. Once the data has been processed, it can be removed from the cache, and it will never be restored. For simplicity, we handle this by using a set of documents that represents "sensed data since the last sync". The logical representation of this data is shown in the Data-interchange-specifications document. The expectation is that this data will be dumped directly into the event database upon arrival at the server with minor formatting changes imposed.
3. Messages: Events generated by the system, may be based on user input or system state. Will happen infrequently, and (in production), can be deleted once they are processed. (Will we delete them? Or are at least the transitions useful?)
4. UI screens: We also want to deploy UI screens to the phone filesystem in order to support dynamic, on-the-fly updating without having to go through the app store. We will defer the design of this component until the first two are implemented, but a high-level solution might be to send a set of git URLs which can be cloned onto the phone filesystem.

It is clear that our needs fit into several common systems patterns, but need to be adapted for offline operation. In particular, users should have the option to delay UI screen deployments and sensed data uploads until the phone is connected over WiFi.

Comparison to current architecture

The most popular current commercial phone app architectures, exemplified by Meteor, is to use the phone largely as a display terminal. This implies the following flow: gather data from user input -> analyse/share on server -> send push notification -> load and display result when user launches the app. This does not take into account the full capabilities of the phone as a computation device, capable of working in the background. Part of the reason for this might be that this is the mode that iOS favors - background operation is severely restricted and is only permitted for a small set of operations. But this really restricts the potential of the phone as a computation device. We do not have this restriction since our app is location centric and is permitted to run in the background, even on iOS. This allows us to explore a more rich computational experience on the phone with periodic syncs of information and more independent operation. This also avoids having the radio on all the time, which is a significant battery drain. It is unclear if the CPU or the radio is the bigger drain, but given that both the CPU and the radio have to be turned on to send or receive information, having the radio off seems to be a net win.

Data organization

The primary design decisions for the data organization of the user cache deal the structure of the data.

1. One option is to structure the data according to source and function. An example might be to separate data generated by the user versus data that is sensed automatically, or data that is displayed to the user versus data that is used to influence background operation.
2. A second option is to structure the data in a format similar to existing caches in operating systems such as memory and disk caches. In this option, the cache does not know the state of its contents. Each entry consists of metadata, and interchangeable data. The cache does not attempt to interpret the data. The data can be swapped in and out at will and transparently by the cache.

In general, we assume that option (2) is better because it is more standard and has an extensive literature.

Our decision

In our case, because of our set of requirements, we will use a hybrid approach in which our cache entries are agnostic to the data that they store, but the metadata provides some indication of the data's place in the logical structure. The logical structure, with examples, is shown in Data-interchange-specifications.

1. Read-only documents: Documents cached on the phone for offline access. Typically updated or replaced with a new version, only deleted if the phone code no longer needs to access it. Each document typically has a unique key that represents the path to it in the logical representation of the data.

2. Sensed data: Data sent from the phone to the server, can be deleted from the cache after processing. Multiple entries for the same key, can be grouped and send to the server in a batch.

3. Messages: Data sent from the phone to the server, can be deleted after processing. Examples include state transition, reboot events, etc. As a rule of thumb, events read directly from the system such as locations or accelerometer data are sensor events. Events inferred by our code, such as transitions, are not. This can be a bit tricky, based on the definition of "system". For example:

   • activity detection runs on top of the accelerometer data to infer activity. But this is done by the Android "system".
   • Similarly, visit detection is done as part of the iOS "system".

   We will treat these as sensor data as well, since they represent immutable, black box data to us.We will store these and evaluate various algorithms that run on top of them.

4. Files: Details later.

5. Read-write documents: A special consideration is that of read-write documents - documents that the user creates, and which need to be sent to the server, but also displayed to the user. A naive approach might be to assume that the document is sent to the server, processed immediately and the modified information is displayed to the user. However, this is would not take into account offline operation. In order to handle offline operation, the values should be displayed to the user from the cache until the push to server -> analysis -> push to client cycle is complete. This is a fairly simple override pattern, except for the question of when the entries will be deleted or purged from the cache. Let us consider two examples to motivate the answer.

• confirmed modes: confirmed modes can be purged as soon as the corresponding trip in the document cache has the confirmed mode set
• primary car mpg: is never purged - retained even after the server has processed it. Maybe we can say that it is purged after the mpg has shown up in the user's profile. This assumes that the user profile is a read-only document, which makes sense. So we will make the code that does the unification of the data from two sources (written version if exists, read version otherwise), purge the cache since it already knows which write version and which read version map to each other.

Implementation version

In keeping with (2) above, our cache will store entries without attempting to interpret the data. This means that information on interpretation needs to be stored in metadata.

The metadata for each entry consists of:

1. write_ts: The timestamp at which the entry was last written
2. read_ts: The timestamp at which the entry was last read. Currently not implemented.
3. type: 'document', 'rw-document', 'sensor-data', 'message' or 'file'
4. key: e.g. 'data/game' or 'background/accelerometer'
5. platform: 'android' or 'ios'. This helps us parse the data correctly on the server. Putting this as part of the metadata allows us to support
6. plugin: an indication of the plugin that the document is associated with. Might be useful for documents, to indicate when they can be purged from the cache - a document related to a plugin that is no longer in use can be safely purged. Currently not implemented.
7. weight: "light" to represent small and quick data that can be transferred over all types of network; "heavy" to represent heavyweight data, such as user screens, that should be preferentially be transferred only over WiFi (if the user specifies). Currently not implemented.

In addition, each entry has a data field that contains the data in JSON format.

Here is an example for each type of entry.

1. Read only document:

   {
     "metadata": {
       "write_ts": 1435856137,
       "read_ts": 1435856138,
       "type": "document",
       "key": "data/carbon_footprint",
       "plugin": "data"
     },
     "data" : {
       "mine": 45.64,
       "avg": 21.35,
       "optimal": 44.21
     }
   }
   

2. Message:

   {
     "metadata": {
       "write_ts": 1435856237,
       "read_ts": 1435856238,
       "type": "message",
       "key": "statemachine/transition",
     },
     "data" : {
       "curr_state": "ongoing_trip",
       "transition": "detected_trip_end",
     }
   }
   

3. Sensor data:

   {
     "metadata": {
       "write_ts": 1435856237,
       "read_ts": 1435856238,
       "type": "sensor-data",
       "key": "background/location",
     },
     "data" : {
       "mLat": 45.64,
       "mLng": 21.35,
       "time": 1435856237,
     }
   }
   

4. Read/Write document:

   {
     "metadata": {
       "write_ts": 1435856237,
       "read_ts": 1435856238,
       "type": "rw-document",
       "key": "diary/mode_confirmation",
     },
     "data" : {
       "section_id": sid1,
       "mode": "walking",
     }
   }