Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Malformed cursor causing issues with paginated queries #362

Open
ngomezdelcampo opened this issue Mar 4, 2024 · 3 comments · May be fixed by #368
Open

Malformed cursor causing issues with paginated queries #362

ngomezdelcampo opened this issue Mar 4, 2024 · 3 comments · May be fixed by #368

Comments

@ngomezdelcampo
Copy link

ngomezdelcampo commented Mar 4, 2024 •
edited
Loading

Describe the bug
When I use a query the count option, my cursor is sometimes malformed. Note that the merchantid is incomplete.

eyJwayI6IiRwYWxvbW1hI21lcmNoYW50aWRfIiwic2siOiIkY3VzdG9tZXJfMSNpZF8wMWhwancwOGU0Y2c4anIyOGNkZnEzdzhqOCJ9
{"pk":"$palomma#merchantid_","sk":"$customer_1#id_01hpjw08e4cg8jr28cdfq3w8j8"}

When I make the same query with the limit option, the cursor is properly formed.

eyJwayI6IiRwYWxvbW1hI21lcmNoYW50aWRfZmVsaXBlIiwic2siOiIkY3VzdG9tZXJfMSNpZF8wMWhwaHB2bnNod2RlYTd4MzJxZWc3MzljayJ9
{"pk":"$palomma#merchantid_felipe","sk":"$customer_1#id_01hphpvnshwdea7x32qeg739ck"}

The impact is that pagination breaks with count with the following error:

The query can return at most one row and cannot be restarted

ElectroDB Version
2.13.1

ElectroDB Playground Link
https://electrodb.fun/?#code/PQKgBAsg9gJgpgGzARwK5wE4Es4GcA0YuccYGeqCALgUQBYCG5YA7llXWAGbZwB2MXGBDAAUKKwBbAA5QMVMAG8wAUT5V2AT0IBlTADcsAY1IBfbhiiSwAIkRwjVSzABGNgNzijUPrgVUGFwRSAF5bABVA4I9xUDAAVWIMITUNKk1hMW9fBVQkoTC+OBZVdS0AClEwJSrqsElYRAAuGrq6-jTNFpsjPKorTBt8WrakwxNuseM4IZG6-UxcLB9ugEZZttNhtoYqJywXVCo8FsU56slMI0Z1AEkYU-O69Ok4Sf2+AHMNtrryNCw5AeYCc6G2v2qdCwMHgfBaoLg4M2SLq0MeEOqLzetj82C+Pwh-1QgLgwIRKLa5AYMAA8nwEF0QRgwU9qvAuAxKFQWuUAJRgEIAPjARgwmmk-QAdBgGAIrPF4rcACJ8p5bJ7kLiYfgTVoYrHvPHfClgdUQvgMS7o-Xi7E2XFYfEm6pEklk5mItUmmBQXqXdThW3WiEGsAAbR6RiGthM0ZsfHYNgAumAGEJsn5nWQ4ACgfCPSazb8fX6OgA5VCSFyYYO-UP2j7G1nZ3Ok-MsiFFtpwSQMLAIWttesOp3N115pkd35durSOg+OAVqs1vUhoM4xsE37jtuTz2dk2iuC7UkAQW5q7r64bRq3lOPtPpjPJY5zxInL4xYHZnOoPP5QpgEUJRKiefKSv0tw6DSOiNnyhYmqg0gwCeMDnoOzzXiOTZfiwuzXN0IB3n8b5uu2+4Yj+XL-gKwrAWAoHHOBkHQbBRrwc2xAXuUSEoccaFUABgrNtUvGoeeYAAGSSWAACEWC4GWDBluUjFwJK0hMMQPHIeJgm8iJdQAPxgGJ-HnoZ1QtPRanMVAUEwXBvIIXMM6OvAAAeJyXtULiaBAVw3FQ9wYdU0gANanNwOAIMCNgRdG3gyFASzHC0EaXBg1yysFMDJqaWa4JFSjRYgcVFYlViyKl2IRtC+UztUjVgH5ABKcBauQfC6mcX4RVFXAxXFnxLAALAlhBJdV7C1TYmXZXceUps11RFQNQ3dCNWCjRVk1VSlM3pTYmrat1MzLVm7lwB5m1jcRM5FkWygBEEZiiLyniiHkiySkeYG9fUgU5SFtgJt4PwWlatgAApwNI7BQGAsPkAAXj8JaVh0gavN0RhRuCGP+lQS7VhgawAEwAMyjQArAAbAA7AAHAAnD8PZ9gOtivPD-QAAKfL2-a-VYPxzguJMrjYlOrLLctyzYoimLykqfFAfKeN9yS-VSTEA-NQUg-GxhQBDlp2rDvOI8jcBowTvqYwG154+jDtE5LZO2KsVO04zrPs0LXPxXDCMC4HIuSGL85FB73Qy-L8uK8rqvqx9X35Drx567UBvA3FYOm+CkMWyH-RI9qdu1ITWPO-jVdu+WlakxT1P08zbPghz-bdDzoeC5zEdRxLTdS-HCey0nKtqxr6c-X92cXEDi3dAXZtQzYlsI+XqOu6WTs4zGddsg36ix17Ptt-7neBz3pdQGHA9JUPMcj570vjwrStT6nn3AMAplLC+CKKAqB1CEA4KQYg2QYAoHQGKFqVJwq4FntrNAmBNCSj8gFLKhsYDlEULnZeoMTY2GTtPAGbRvCgIvOTE0ch4Bv3gLgI+dRlaawzmgsUmD-JL1yvgwhuUV4kLIerChdQqHqBaLQp4vRkhyG6NGP+qY8LsBBHQUgg1kgKE4RkWUMDpBHDUaQWRuA5BgEdGAdR5Anj0KlkwlhTU06xH-nkR0nwwAICkOwcB6iiAOB8DAnR5ihCDSKCg3AkodHcOwQtPhBDeFG1XiI-BTxPGSHYFIuhGAGHdHsVuNh4TIlwIwVghJeD4k4LzkI8GySxHVDSRksA0iIQmPkbYRR-8GAqP8L4zRfhYHoNTAIMABienGNQHIjA5i+CWO1DY7Jdi8AONNB9IAA

Entity/Service Definitions
Include your entity model (or a model that sufficiently recreates your issue) to help troubleshoot.

  {
    model: {
      entity: "customer",
      service: "service",
      version: "1",
    },
    attributes: {
      merchantId: {
        type: "string",
        required: true,
        hidden: true,
      },
      id: {
        type: "string",
        required: true,
        readOnly: true,
        default: () => ulid()
      },
      reference: {
        type: "string",
      },
      name: {
        type: "string",
        required: true,
      },
      documentType: {
        type: ["cc", "ce", "nit"] as const,
        required: true,
      },
      documentNumber: {
        type: "string",
        required: true,
      },
      email: {
        type: "string",
        required: true,
      },
      phoneNumber: {
        type: "string",
        required: true,
      },
      createdAt: {
        type: "string",
        readOnly: true,
        required: true,
        default: () => new Date().toISOString(),
      },
      updatedAt: {
        type: "string",
        watch: "*",
        required: true,
        default: () => new Date().toISOString(),
        set: (updatedAt) =>
          updatedAt && !isNaN(Date.parse(updatedAt))
            ? updatedAt
            : new Date().toISOString(),
      },
    },
    indexes: {
      byMerchantId: {
        pk: { field: "pk", composite: ["merchantId"] },
        sk: { field: "sk", composite: ["id"] },
      },
      byReference: {
        pk: { field: "gsi4pk", composite: ["merchantId"] },
        sk: { field: "gsi4sk", composite: ["reference"] },
        index: "gsi4",
      },
    },
  }

Expected behavior
The cursor of the query operation with the count option should give a properly formed cursor that enables me to query the next page.

Errors

The query can return at most one row and cannot be restarted
@tywalch
Copy link
Owner

tywalch commented Mar 27, 2024

Thank you I will look into this!

tywalch added a commit that referenced this issue Mar 28, 2024
@tywalch
Fixes 362
a6d6c9e 
This branch fixes issue #362. The root cause of the issue was that one of the composite attributes was hidden. Because the cursor was created using the "formatted" return item, it was the hidden composite was not available to the cursor formatting function. We now check for this unique case and perform some additional calculations. Ideally this reduces the surface area of performance implications to only entities that have a hidden composite attribute defined.
@tywalch tywalch linked a pull request Mar 28, 2024 that will close this issue
Open
@tywalch tywalch linked a pull request Mar 28, 2024 that will close this issue
Fixes 362 #368
Open
@tywalch
Copy link
Owner

tywalch commented Mar 28, 2024 •
edited
Loading

@ngomezdelcampo I started a PR request for this, I still need to add unit tests but I have identified the root cause: The composite attribute merchantId is flagged as hidden which has internal ramifications preventing it from being included at cursor creation.

@tywalch
Copy link
Owner

tywalch commented May 1, 2024

@ngomezdelcampo Small update on this: I started working on this PR and found that the underlying implementation was very disagreeable with this scenario. It proved a bit difficult to not add more overhead to react differently in this case, and I think it will require a bit more time determine a better refactor. Can you confirm whether or not you are still in need of this fix, or if you have moved on? I am working to triage some recent issues, thank you for your patience so far

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixes 362 tywalch/electrodb
2 participants
@tywalch @ngomezdelcampo