Skip to content

Commit

Permalink
docs: Improve store internals documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
@paultranvan
paultranvan committed May 30, 2024
1 parent 2ded951 commit 3855662
Showing 1 changed file with 172 additions and 41 deletions.
213 changes: 172 additions & 41 deletions docs/architecture.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ UI components to the data.
advanced techniques where you create selectors directly on the data
of cozy-client.

The redux store is composed of two `slices` : `documents` and `queries`:
The redux store is composed of two collections: `documents` and `queries`:

- `documents` stores the data indexed by [doctype](https://docs.cozy.io/en/cozy-doctypes/docs/) then `_id`
- `queries` store information for each query that has been done by cozy-client.
Expand All @@ -112,71 +112,199 @@ The redux store is composed of two `slices` : `documents` and `queries`:

}
```
> ℹ️ If queries are not named, a name is automatically generated but this means that the `queries`
slice can grow indefinitely if there are a large number of queries. This is why you are
encouraged to name your queries : `client.query(qdef, { as: 'finishedTodos'})`.
> ⚠️ If queries are not named, a name is automatically generated, but this means that the `queries`
collection can grow indefinitely. This is why you are encouraged to name your queries : `client.query(qdef, { as: 'finishedTodos'})`.

The glue between the Redux store and the UI is done via ObservableQuery.
`ObservableQuery` are objects instantiated by a <Query /> component. Their
role is to react to store changes and wake the component. They should not
be used directly as `useQuery` and `queryConnect` do the job for you.
ℹ️ See the [react integration](/docs/react-integration.md) for more insight about the glue between the redux store and the UI.

What do we mean exactly by saying "This redux store
brings observability to cozy-client, and allows for connection of
UI components to the data." Let's take a full exemple:

We have a component that displays a todolist:
Here is a simple example on how to display data with cozy-client:
```jsx
// TodoListComponent.jsx
const { data, fetchStatus } = useQuery(Q('io.cozy.todos'), {'as': 'todoslist'})
if(fetchStatus === 'loading'){
return <Spinner />
}
if(fetchStatus === 'loaded'){
return <TodoLists todos={data}> />
}
```

But we also have a component that gives use the opportunity to add a
new todo:
```jsx
client.save({_type: 'io.cozy.todos', 'label': 'New TODO'});
This way, a spinner will be displayed during the time the query is actually run. Once the query is done, data is available and can be displayed.

Any data change will be automatically handled by cozy-client.


### Queries

Here, we describe what happens internally when a query is called from an app.

First, it is important to understand that each query is stored in the store, in a `queries` associative array. Each query has a unique id, a set of metadata, and the list of retrieved document ids.

For example:
```js
{
query1: {
definition: {
...
},
fetchStatus: "loaded",
lastFetch: 1716989816939,
lastUpdate: 1716989816939,
hasMore: false,
count: 2,
bookmark: "xyz",
data: ["docId1", "docId2"] // confusingly named `data`, but only stores ids
}
}
```

After the call to `save` your `TodoListComponent` will be re-rendered
with the `New TODO`.
#### Fetch status

An important attribute used for the query lifecycle is the `fetchStatus`.
It can take the following values:
- `pending`: the query is about to be run for the first time.
- `loading`: the query is currently running and might return results. From a UX perspective, it is often used by apps to display a spinner.
- `loaded`: the query had been run and returned results.
- `failed`: the last query execution failed.

When a query is called for the first time, it is first initialized in the `queries` collection in the store, with the `pending` status.

If the query already exists in the store, its status is checked to ensure that it is not already in a `loading` state.

Then, the query is "loaded", making its status on `loading`.

### How does it works?
Finally, the query is actually run and the results are retrieved from the database. The status is now `loaded` and some additional information are saved in the query store, such as `lastFetch` and `lastUpdate`.
If any new data is retrieved by the query, all the documents ids retrieved by the query are then stored in the `data` array.

When a `query` is resolved, CozyClient dispatches a `receiveQueryResult`
action for a simple `get` but `receiveMutationResult` when we mutate
something.
Likewise, the `documents` collection is updated as well with the full documents content.

### Focus on `receiveMutationResult`:

Our two slices `documents` and `queries` listen actions and do some specific work on
`isReceivingMutationResult` action.
#### Auto query update

`documents`: If the `_id` of the mutated document is not present, then we add the document.
If the `_id` is already there, then we update the content of the document with the fresh data
(for now the work is done in extractAndMergeDocument() method).
Since there is a link between queries and documents through the ids list in each query object, we need to be able to deal with changes occuring in the documents collection, i.e. when documents are added or deleted.

So if your app is linked to the documents store via getDocumentFromStore() for instance
your app will have the updated value.
For this, cozy-client has a mechanism called "query updater": each time a query is run, all the queries in the store are re-evaluated directly in the store.
That means that for each query, we take its definition, convert it into a mango predicate, and run it against the store's document collection.
For each query result, we check if there is any change in the known ids list, and update it accordingly.

`queries` has an autoUpdater mechanism that does something we can explain this way:
- it takes the mutated documents (newly created or updated)
- it converts the initial `query` to a "`js predicate`" (thanks to the [sift library](https://github.com/crcn/sift.js/))
- For each query already in the `slice` it runs this `js predicate` and detects if the query
is concerned by the mutation
- If the query is concerned, then it checks if it has to remove / add the id of the `mutated`
document
(for now the work is done mainly in queries.js/updateData())
Thanks to this, we ensure that all the queries are always up-to-date, and apps can be effectively rendered thanks to the [react integration](/docs/react-integration.md).

So in our previous example our `todoslist` is concerned by the addition of the new todo, then
the `id` is added to `todolist` data, then the component linked will be refreshed with this new
document.

### About updates
#### Fetch policy and query naming

As already mentioned, when a mutated document already exists in the store throught its `_id`, it is updated with the new content. However, this update never removes existing fields, it only merges or adds new fields. This is typically useful when two queries retrieve the same document, but not with all the fields.
The fetch policy is a simple mechanism useful to prevent excessive fetching.
It is optionnaly defined for each query and define how many time should be elapsed since the last query execution before running again the query.

In this example we query the checked todos, only if the last execution occured more than 10 minutes ago:
```js
const queryDef = Q('io.cozy.todos').where(checked: true)
const queryOptions = {
as: 'checkedTodos', // query unique name, to find it in the store
fetchPolicy: CozyClient.fetchPolicies.olderThan(10 * 60 * 1000) // fetch policy, for 10min
}
await client.query(queryDef, queryOptions)
```

If the query had been run before the 10 minutes, an early return will happen and nothing will be updated in the store.

💡 Thanks to the auto-query updater, the data will not be stale if there is a data change in the app: indeed, any change on the documents will trigger the in-memory re-evaluation of the query and a render on the app if something is different.

ℹ️ The `as` option is used to name the query, which is always a good idea to ensure that it has a unique name. When it's not specified, a random id is generated, resulting in potential query duplication in the store. Likewise, one must be very careful to prevent queries with the same name, as it would mixup results in the store.


#### Complete query flow

```mermaid
---
title: Query with cozy-client
---
sequenceDiagram
participant App as App
participant CC as cozy-client
participant Store as Store
participant Backend as Backend
App->>CC: Query documents
CC->>Store: Check query
alt Query does not exist
CC->>Store: Init query
Store->>Store: Add query to collecton
else Query exists
CC->>CC: Check fetch policy
CC->>CC: Check loading status
end
CC->>Store: Load query
Store->>Store: Update query status
CC->>Backend: Execute query
Backend->>CC: Data
CC->>Store: Data
Store->>Store: Update query status
Store->>Store: Update documents
CC->>App: Results
Store->>Store: Auto-update queries
```

#### Persisted store and background fetching stategy

An app can decide to persist its store in localstorage. See [here](https://github.com/cozy/cozy-home/blob/c47858515d4f95c24fa88a6b96bb0b500b947424/src/store/configureStore.js#L46) for an example.
When doing so, the app loads the store when it starts and benefit from the instant available cached data.

But, the data could have significantly evolved between the last store save and the start, making the documents potentially stale.
Hopefully, since the fetch policy is probably expired, queries will be run again, retrieving fresh data. Unfortunately, this might cause an immediate spinning display on the app when it starts, even though there is available data thanks to the persisted store.

To solve this, a `backgroundFetching` option can be set, at the query level, or at the store level (enabled for all queries).

When the query is loaded, it simply keeps the query status in `loaded` state (rather than setting a `loading` state) and set a new `isFetching` attribute, stored in the query.
Thanks to this, the app can adapt its display to avoid the "spinning effect", and might inform the user that data is being updated in the background, without refreshing the whole app.

See [this PR](https://github.com/cozy/cozy-client/pull/1211) for more insights.


Here is a simplified execution sequence with or without background fetching:
```mermaid
---
title: Background fetching
---
sequenceDiagram
participant App as App
participant CC as cozy-client
participant Store as Store
participant Backend as Backend
App->>CC: Query documents
CC->>Store: Load query
alt Without background fetching
Store->>Store: update query status
Store->>App: Loading query status
App->>App: Spinner
else With background fetching
Store->>Store: no change on query status
Store->>App: isFetching status
App->>App: Do nothing
end
CC->>Backend: Execute query
Backend->>CC: New data
CC->>Store: Update store
CC->>App: New data
App->>App: Refresh display
```

### Updates

Here, we detail what happens when a document is updated, i.e. a mutation happens, typically after a `client.save`.

If the document `_id` does not exist in the store's `documents` collection, i.e. after the a creation, it is simply added.
If it already exists, the document content is updated in this collection.

Furthermore, just like with the queries, any mutation triggers the [auto query updater](#auto-query-update), that can result in adding or removing document ids.

Thanks to it, the app is guaranteed to have the up-to-date documents, returned from the `documents` collection.

⚠️ As already mentioned, when a mutated document already exists in the store throught its `_id`, it is updated with the new content. However, this update never removes existing fields, it only merges or adds new fields. This is typically useful when two queries retrieve the same document, but not with all the fields.
Let us illustrate with this example:

```js
Expand All @@ -190,3 +318,6 @@ await client.query(Q1) // returns { _id: 'my-todo', _type: 'io.cozy.todos', labe

Q1 retrieves the full content of 'my-todo' while Q2 only retrieves its label. When Q2 is run, 'my-todo' already exists in the store, as Q1 was previsouly run. However, the fields retrieved by Q1 for 'my-todos' but absent from Q2 are not lost. This way, when Q1 is run again, the results fetched from the store (thanks to the fetch policy) will include all the fields.

The problem is when some fields are removed from a document: the store will not reflect this change and result in stale data.


0 comments on commit 3855662

Please sign in to comment.