Corrupted database where a document has 2 root revisions

[nono]

Description

I don't know what happened, but I have a database where some documents are in a very weird state. Let's take one of those documents:

$ curl -s "http://couch:5984/cozyxxxxxxxxxxxxxxxxxxxxxxx%2Fio-cozy-files/78ca2b76808dd591f1b64f1f137bfe48?meta=true&revs=true&open_revs=all"
--41adad57f1b406a22332b1bc52b378c4
Content-Type: application/json

{"_id":"78ca2b76808dd591f1b64f1f137bfe48","_rev":"5-bc7fe0c9fec74c7d857d17e97121aebe","cozyMetadata":{"doctypeVersion":"1","metadataVersion":1,"createdAt":"2020-01-10T17:18:59.681880993Z","createdByApp":"cozy-desktop","createdByAppVersion":"3.17.0","updatedAt":"2020-01-10T17:20:02.966474161Z","createdOn":"https://xxxxxx.mycozy.cloud/"},"created_at":"2020-01-10T17:18:59Z","dir_id":"3ec8cdd199d7081a9361f46c22615721","name":"[REDACTED]","path":"/[REDACTED]","tags":[],"type":"directory","updated_at":"2020-01-10T17:18:59Z","_revisions":{"start":5,"ids":["bc7fe0c9fec74c7d857d17e97121aebe","ed6c725a3249ce90ce4155d0180d3c07","9e0c917a7719b181293c22207a54e195","cf252a1dfa81712e140c3b9b2fa25347","90d8ea315491928e8f3199360a0a48bd"]}}
--41adad57f1b406a22332b1bc52b378c4
Content-Type: application/json

{"_id":"78ca2b76808dd591f1b64f1f137bfe48","_rev":"11-9236dbe8f2baf8e5a1384fa64bd61545","_deleted":true,"_revisions":{"start":11,"ids":["9236dbe8f2baf8e5a1384fa64bd61545","62ceec92c6557ab036ec6a808dba8b8a","b5c87beae0e88912edcfab866d32fd6f","a0e1231615202ba37adb60179e8373d0","e7cadf01aae5cab26d71b6e7405f0e35","2a0151e2c632e15f7584d94ea2520449","bc7fe0c9fec74c7d857d17e97121aebe","ed6c725a3249ce90ce4155d0180d3c07","9e0c917a7719b181293c22207a54e195","cf252a1dfa81712e140c3b9b2fa25347","845faf29d2ab2d455c8f47948062676b"]}}
--41adad57f1b406a22332b1bc52b378c4--

CouchDB tells me that there is a conflict on this document, with 2 branches. The branch for 11-9236dbe8f2baf8e5a1384fa64bd61545 has lost the conflict, as it is marked as _deleted. And the other branch ends with 5-bc7fe0c9fec74c7d857d17e97121aebe.

But, when looking at the _revisions, we can see that 5-bc7fe0c9fec74c7d857d17e97121aebe is an ancestor of 11-9236dbe8f2baf8e5a1384fa64bd61545. So, there should be no conflict.

And, the really weird thing: both branches have the same revisions 5, 4, 3, and 2, but their first revisions are not the same: 90d8ea315491928e8f3199360a0a48bd and 845faf29d2ab2d455c8f47948062676b. I don't understand how the internal structures of CouchDB for storing the revisions tree can give us two roots.

By the way, I also got:

  "_revs_info": [
    {
      "rev": "5-bc7fe0c9fec74c7d857d17e97121aebe",
      "status": "missing"
    },

I think that this revision was compacted on some nodes of the cluster, but not on others.

Steps to Reproduce

I don't know how to reproduce. I understand that without a way to reproduce, it would be very hard to fix the bug. For the moment, I'd like to understand how it is possible that CouchDB sees two roots in a revision tree, and why a revision that is ancestor of another revision can be seen as a leaf.

Expected Behaviour

Well, first, I would have expected that CouchDB enforces that the revisions for a document are a tree. And, that a revision listed in _revisions for another revision cannot be seen as a leaf. And that leaf revisions cannot be missing (compacted).

Your Environment

• CouchDB version used: 2.3.0
• Operating system and version: Debian 9.11

{"couchdb":"Welcome","version":"2.3.0","git_sha":"07ea0c7","uuid":"b3ac19e755fb8720ec0be49b61086e87","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}

Additional Context

To fix a document, we can go this way:

1. Add a new revision to the _deleted branch
2. Compact the database
3. Purge the old revisions
4. Delete again the document

But it's a manual operation. And it doesn't explain how we were in this situation.

[davisp]

Hiya @nono,

This looks like either something was creating its own revisions or there was an accidental copying of revisions between documents that were then inserted using new_edits=false. A subtlety of the tree is that the the revision identifiers only have to be unique at the node level. Generally they're unique to the tree unless we get into shenangians around using new_edits=false in which case the actual constraints of the tree are much looser than what normally happens. And unfortunately a lot of the traversal algorithms assume unique revisions so when you specify them for some operations you may not be getting the one you expect.

Your best bet would be to purge the 11- revision so that you're back to a linear history and then move on. In the future I'd avoid using new_edits=false in application code.

Paul