Custom attribute types and queries on such attributes

[leptonite]

I'll refer to this example when describing my issue below.

For an attribute (line 53) of my item I want to use instances of a custom class (line 4). They should be serialized to a string when writing into the DynamoDB table and deserialized into instances of the custom class when reading the table. Furthermore, EntityItem<...> should infer the custom class rather than string for this attribute (line 78). This works fine by using CustomAttributeType, but my first question is: Did I do it the right way?

Next I want to use this attribute in a query like this: [...].where(({ myAttribute }, { eq }) => eq(myAttribute, new MyCustomClass([...]))).go()

I tried this in three ways (lines 100, 103 and 106) and I expected the version in line 100 to work, but it doesn't. I think it could work, ElectroDB just needs to apply the attribute's set function (line 57) not only to attribute values, but also to query values on this attribute.

Do you agree that ElectroDB should work the way I expected?

If not, how should I write queries on such attributes?

[tywalch]

Hi @leptonite 👋

Great question! We have a few options, but I want to first address why filters don't go down the usual path that calls get, set, and validate. Filtering often requires very precise/deliberate choices on behalf of the caller, for example: with the contains operation you are likely providing a partial copy of the full attribute, so running set or validate on partially provided data could have frustrating consequences for users. As a result, the requirements placed on filter values are lean; only validating paths exist and types match, which brings us to your challenge!

There are a number of ways we can address this, but here are the top three stand outs:

Option 1

Whenever you struggle with a filtering issue, remember that the best part of how the where() clause works is that it returns the actual string that will be used in the ConditionExpression. There are a few utility operations you can use to help in this case, namely name(), and escape(). Here is a playground showing this approach, with a little blurb about what it's doing!

Option 2

The next option involves removing the stringification of your Color class, and filtering on the individual properties. This assumes that the stringified value isn't being used since it's serialized on write and deserialized on retrieval. I assume this option won't work for you, while I am interested in your use-case for serializing the value, but I am sure it's part of a bigger requirement. I mention this as an option though because there is actually a mix of option 1 and option 2 next...

Option 3

Here is another approach for you to consider: the attribute option watch (read more here) was created to make denormalization easier, and I think it might be helpful here!

Quick break for story time: one project I had required me to create a "transaction warehouse" that stored user purchase transactions. There was a need to filter by the transaction description (e.g. filter by the term "starbucks"). Since DynamoDB is case sensitive, and the descriptions were well-formatted, this posed a problem. This is where watch came into play. The watch option allows one attribute to have their get and/or set attributes triggered when another attribute is triggered. So for the transaction description, I created another attribute called filterDescription which watched description, and set its value own value to be a lowercase version of the description with no spaces. I also used the hidden and readOnly options to remove its exposure to the frontend. This allowed for a wildly improved filter experience by users. Here is an example of that in action

How that relates to your use case is maybe you can have your color attribute, and an rgb attribute that watches color. Something like this.

Let me know if this helps, I can go into more detail on any of the above and/or help you brainstorm more ideas based on your use case 👍

[leptonite]

Hi @tywalch,

thank you for your detailed response. My actual use case are the classes Instant and LocalDate from js-joda and hopefully in the not too distant future the classes Instant and PlainDate of the Temporal API. But let's stop the discussion about date/time libs before it even starts and go back to the actual topic of this thread ;)

Besides solving this single use case I want to understand the general concept and limitations of custom attribute types in electrodb.

• Option 1: This will work, but it means giving up type safety in queries and it's a bit inconvenient to write.
• Option 2: This may work, but now that you know my use case you probably understand that I prefer stringification.
• Option 3: This means storing the value twice, doesn’t it?

In the meantime I came across another problem with my custom attribute type: I cannot use it as part of an index. That’s an additional requirement which I didn’t address until now. And it’s another reason for stringification and therefore against option 2.

All this makes me think that I am somewhat abusing CustomAttributeType and the get and set callbacks. I tend to turning this discussion into a feature request:

I want to be able to define a custom attribute type that has the following properties:

• It has a string representation and a runtime representation.
• The user provides functions to convert between these representations.
• electrodb uses the string representation to store the value in the database.
• electrodb uses the string representation to build queries for DynamoDB.
• electrodb uses the string representation to concatenate the value into partition or sort keys (also for GSIs).
• electrodb’s API always uses the runtime representation which means the user only works with the runtime representation.
• electrodb uses the provided callbacks for conversion from and to string.

This may look like this:

import {createCustomStringifyableAttributeType} from electrodb;
// createCustomStringifyableAttributeType<T>(stringify: (runtimeRepr: T) => string, parse: (stringRepr: string) => T) => ???

// create a new attribute type and tell electrodb how to convert it from and to string
const colorAttributeType = createCustomStringifyableAttributeType(stringifyColor, parseColor);

const coloredThing = new Entity(
  {
    model: { /* ... */ },
    attributes: {
      id: {
        type: 'string',
        required: true,
      },
      name: {
        type: 'string',
        required: true,
      },
      color: {
        type: colorAttributeType, // use the new type. get and set callbacks not needed because colorAttributeType knows how to convert
        required: true,
      },
    },
    indexes: {
      byId: { /* ... */ },
      byColor: {
        pk: {
          field: 'PK',
          composite: ['color'], // use custom attribute types in an index, electrodb automatically converts into the string representation
        },
        sk: {
          field: 'SK',
          composite: [],
        },
      },
    },
  }
);

type ColoredThing = EntityItem<typeof coloredThing>

const lemon: ColoredThing = {
  id: 'lemon1',
  name: 'lemon',
  color: new Color(255, 255, 0),
}

coloredThing.put(lemon).go()

const renamedLemon: ColoredThing = {
  id: 'lemon1',
  name: 'lemon with new name',
  color: new Color(255, 255, 0),
}

coloredThing.put(renamedLemon).where(({ color }, { eq }) => eq(color, new Color(255, 255, 0))).go() // use custom attribute type in queries. electrodb knows that literal values which are compared to the color attribute must be converted with stringifyColor

I am aware that this feature would be somewhat incomplete:

• It can only convert to and from string, but there may be custom attribute types which are better converted to e. g. binary or object. However, strings are the most flexible because they can be used as part of a composed index attribute. Therefore I think the string version is the most important one.
• There may be use cases for some typesafe prefix quieries. For example lets say we have two custom stringifyable attribute types, one for js-joda LocalDate (2024-03-04) and one for js-joda YearMonth (2024-03). With a LocalDate attribute as part of a sort key we may want to query like entity.query.byLocalDate({}).begins({localDateAttribute: YearMonth.of(2024, 3)).go(). Something like this is probably tricky to support.

I'm curious about your opinion on this.