Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VO-874] feat: Improve index naming #1495

Merged
merged 2 commits into from
Jul 8, 2024
Merged

Conversation

cballevre
Copy link
Member

@cballevre cballevre commented Jul 1, 2024

Index naming has two weaknesses:

  • it doesn't include partial index values

This can cause differences between the expected result and the result when modifying queries.

An example of the new index name:

Query definition :

Q('io.cozy.files')
  .where({
    type: 'file',
  })
  .partialIndex({
    class: {
      $or: ['image', 'pdf'] 
    }
    trashed: {
      $eq: false 
    }
  })
  .sortBy([
    { name: 'desc' }
  ])

Old index name : _design/by_type_and_name_filter_class_and_trashed
New index name : _design/by_type_and_name_filter_(class_(image_$or_pdf))_and_(trashed_$eq_false)

@cballevre cballevre force-pushed the feat/improve-index-naming branch 3 times, most recently from 0af08c5 to 80b3052 Compare July 4, 2024 07:47
@cballevre cballevre marked this pull request as ready for review July 4, 2024 08:00
@cballevre cballevre requested a review from Merkur39 as a code owner July 4, 2024 08:00
Copy link
Contributor

@paultranvan paultranvan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a really great and long-expected work, thank you @cballevre!
My main concern is about the explicit operators, I feel like at least one of your test is not working at it should on couchdb. We might need to push tests a bit further, directly on couchdb

packages/cozy-stack-client/src/DocumentCollection.js Outdated Show resolved Hide resolved
packages/cozy-stack-client/src/mangoIndex.js Outdated Show resolved Hide resolved
packages/cozy-stack-client/src/mangoIndex.js Outdated Show resolved Hide resolved
docs/api/cozy-stack-client.md Show resolved Hide resolved
$or: ['konnector', 'worker']
}
}
expect(makeOperatorsExplicit(query)).toEqual(expected)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you tested this one on couchdb? I don't think that would work like this, I would expect this from couchdb:

  "$or": [
    {
      "type": {
        "$eq": "konnector"
      }
    },
    {
      "type": {
        "$eq": "worker"
      }
    }
  ]

Copy link
Member Author

@cballevre cballevre Jul 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well found it was a failure. I've corrected the function to take this case into account 🙂

Copy link
Contributor

@paultranvan paultranvan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything looks good, but it's probably worth to make some real partial filter testing on couchdb, to ensure we didn't miss anything like the one with the $or

@paultranvan
Copy link
Contributor

Related to #1374

@cballevre cballevre force-pushed the feat/improve-index-naming branch 2 times, most recently from 3fee032 to 1d0a959 Compare July 4, 2024 17:46
The partialFilter values are static. As this does not change, we
decided to integrate them into the index naming. Before this change,
the index was not recalculated when a value changed. So the query
always returned the same result.
When CouchDB creates an index with a partialFilter, it adds explicit
operators when they are implicit, typically the $and and $eq operators.
This causes a mismatch when comparing the partialFilter from the
request definition with the partialFilter from the index. To address
this, we added a step to make the operators in the partialFilter from
the request definition explicit.

This makes it possible to migrate indexes after they have been
renamed rather than re-creating them, which has a non-negligible cost.
@cballevre cballevre changed the title feat: Improve index naming [VO-874] feat: Improve index naming Jul 4, 2024
@cballevre cballevre merged commit 5da0d55 into master Jul 8, 2024
4 checks passed
@cballevre cballevre deleted the feat/improve-index-naming branch July 8, 2024 06:47
@Crash--
Copy link
Contributor

Crash-- commented Jul 15, 2024

So much cleaner than what I did before 👏 👏

Since you have changed the index name, you're going to have a pretty heavy DB server load when you release an app with this (mainly app with services ;))

In fact you're just going to update the name of the index, it should be ok.

return `${indexName}_filter_(${makeKeyFromPartialFilter(partialFilter)})`
}

return indexName
Copy link
Contributor

@Crash-- Crash-- Jul 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You didn't change the method for the Pouch Link. Is it wanted? Is it out of scope ?

Copy link
Contributor

@paultranvan paultranvan Jul 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since @Ldoppea is currently heavily working on pouch link and notably on this part, we didn't feel the need to put it in this PR scope. But this is indeed identified 😉
It also raises a question about the need the mutualize such parts between stack-client and pouch-link... Maybe a new CC package?

@paultranvan
Copy link
Contributor

Since you have changed the index name, you're going to have a pretty heavy DB server load when you release an app with this (mainly app with services ;))

Since we migrate the existing index through a copy, we should avoid a too much consequent load (i.e. we don't recompute index)

@Crash--
Copy link
Contributor

Crash-- commented Jul 15, 2024

Yep, I updated my comment just before yours ;).

Just a last thing: I know that we had conflicts between the sack and cozy-client about index name. A few users had blank page because of this. I think that stack use partialFilters on some place, I'll double check that ;).

@paultranvan
Copy link
Contributor

A few users had blank page because of this. I think that stack use partialFilters on some place, I'll double check that ;).

If you find any info about this, that would be precious!

Ldoppea added a commit that referenced this pull request Jul 22, 2024
In #1495 we improved index naming for CozyStackClient

This commit applies the same algorithm to CozyPouchLink
Ldoppea added a commit that referenced this pull request Jul 22, 2024
In #1495 we improved index naming for CozyStackClient

This commit applies the same algorithm to CozyPouchLink
Ldoppea added a commit that referenced this pull request Jul 26, 2024
In #1495 we improved index naming for CozyStackClient

This commit applies the same algorithm to CozyPouchLink
Ldoppea added a commit that referenced this pull request Jul 26, 2024
In #1495 we improved index naming for CozyStackClient

This commit applies the same algorithm to CozyPouchLink
Ldoppea added a commit that referenced this pull request Jul 30, 2024
In #1495 we improved index naming for CozyStackClient

This commit applies the same algorithm to CozyPouchLink
Ldoppea added a commit that referenced this pull request Aug 13, 2024
In #1495 we improved index naming for CozyStackClient

This commit applies the same algorithm to CozyPouchLink
Ldoppea added a commit that referenced this pull request Sep 9, 2024
In #1495 we improved index naming for CozyStackClient

This commit applies the same algorithm to CozyPouchLink
Ldoppea added a commit that referenced this pull request Sep 9, 2024
In #1495 we improved index naming for CozyStackClient

This commit applies the same algorithm to CozyPouchLink
Ldoppea added a commit that referenced this pull request Sep 9, 2024
In #1495 we improved index naming for CozyStackClient

This commit applies the same algorithm to CozyPouchLink
Ldoppea added a commit that referenced this pull request Sep 19, 2024
In #1495 we improved index naming for CozyStackClient

This commit applies the same algorithm to CozyPouchLink
Ldoppea added a commit that referenced this pull request Sep 19, 2024
In #1495 we improved index naming for CozyStackClient

This commit applies the same algorithm to CozyPouchLink
Ldoppea added a commit that referenced this pull request Sep 24, 2024
In #1495 we improved index naming for CozyStackClient

This commit applies the same algorithm to CozyPouchLink
Ldoppea added a commit that referenced this pull request Sep 24, 2024
In #1495 we improved index naming for CozyStackClient

This commit applies the same algorithm to CozyPouchLink
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants