Data deletion endpoints and Acoustic sync

[leplatrem]

This issue is strongly related with the following ones:

• Add DELETE /ctms/{email_id} and soft deletion #60
• Consider differing levels of access #239
• Delete from Acoustic when deleting contact from DB #571
• Inconsistent handling of empty waitlists and newsletters lists when PUTing a contact #733

Context and Problem Statement

The PUT and DELETE endpoints of the API allow to delete data from the database. And with the current model, the Acoustic sync service does not have anyway to detect them.

• the DELETE endpoint deletes contacts from the database (see Delete from Acoustic when deleting contact from DB #571)
• the PUT endpoint allows to replace contact attributes. For the waitlists and newsletters lists fields, replacing them means deleting existing data that is not part of the overwrite (see Inconsistent handling of empty waitlists and newsletters lists when PUTing a contact #733)

We want to avoid situations where data was deleted from CTMS and still persists in Acoustic (see https://mozilla-hub.atlassian.net/browse/CTMS-146)

Decision Drivers

• Complexity: Low → High: how complex is the solution
• Cost: Low → High: how much engineering efforts is required
• Robustness: Low → High: how resilient to failures

Considered Options

1. [option 1]
2. [option 2]
3. [option 3]

Decision Outcome

Chosen option: "[option 1]", because [justification. e.g., only option, which
meets k.o. criterion decision driver | which resolves force force | … | comes
out best (see below)].

Pros and Cons of the Options

Option 1 - Restrict destructive endpoints

• Add attributes to the API token and do not allow everyone to PUT and DELETE (Consider differing levels of access #239)
• Only admin's token would be allowed to do DELETE in the context of GDRP

Cost: Mid, because it requires some development
Complexity: Low
Robustness: Low, because there could still be situations where admins delete contacts from the DB and don't delete them in Acoustic

Option 2 - Remove the PUT endpoint

Replacing data would only be doable via PATCH requests (specifying all fields to be overwritten)

Cost: Low
Complexity: Low
Robustness: Low, same as option 1. The DELETE endpoint could still lead to inconsistencies.

Option 3 - Implement a sync queue

See #571 (comment)

Cost: Mid-High, because it requires some development
Complexity: Low, because it is a common pattern. And it would probably simplify the sync code.
Robustness: High, because every entry in the sync queue would be added/removed using DB transactions.

Option 4 - Sync by uploading CSVs

Export the DB content every X hours, and have scheduled ingestion jobs on Acoustic.

Cost: Low
Complexity: Low, because we could get rid of sync code
Robustness: Mid, because it relies on Acoustic, but not High because we would not have monitoring of the csv ingestion