Preparation for version 3.0

[tywalch]

I am collecting a list of changes for v3.0.0 so that I can address as many breaking changes in one version as possible. Here are a few (in no particular order):

Remove all automatic filtering and key shifting from comparison queries

• Is this a feature or a bug? #408
• Query validation error - gt and lte create erroneous filter expression #228

This approach was ultimately ill advised for the library to try and own. It results in confusion and unexpected results. This would remove any filters automatically added by electro.

Remove conversion properties from Entity

Breaking off conversion from the Entity instance into a utility function will reduce the footprint of the Entity interface

[Maybe] Remove special query limit logic:

The execution option limit does more than just add a Limit parameter to a query, it will paginate until the limit of a query is at least reached. I am labeling this maybe

---

This is the list currently, let me know if you have any additions you'd like to see!

[zirkelc]

Here are a few suggestions I would like to see:

• a flexible way to mock the Entity functions for get, update, patch, etc. (Mock get, create, update, delete #244)
  I know it's possible to mock go, but it doesn't provide enough information about the operation currently being executed: is it get, patch, update...? An idea would be to have static functions on the Entity class for each operation. Then you could simply do jest.spyOn(Entity.prototype, 'get').mockImplementation(() => {}) to mock the get operation.

• getter/setter/validate/default on entity-level rather than only on attribute-level (similar to this Usage of getters and setters for virtual and calculates attributes #383): after the item was read from the database and all attribute-level functions were applied, it would be good if we had a way to transform and validate the item into another shape. For example, merge multiple separate attributes into one object. So the getter/validate on entity-level should also be able to return different or stricter type.

• remove properties by setting them to null (Feature: use `set({ property: null })` to remove optional properties #399)

• Entity.parse() on Service (Dynamo Stream Handler - Semantics Help #242): I have one dynamodb stream handler which receives events for multiple entities. It would be good to call Service.parse() to return the Entity and/or it's parsed data. I have currrently implemented it like this (pseudo):

type Identifier<S extends Service<any>> = keyof S['entities'];

const rawItem =
  record.eventName === 'REMOVE'
    ? unmarshall(record.dynamodb?.OldImage as DynamoDBImage)
    : unmarshall(record.dynamodb?.NewImage as DynamoDBImage);

const identifier = '__edb_e__' in rawItem ? (rawItem['__edb_e__'] as Identifier<typeof service>) : undefined;

const entity = service.entities[identifier] as Entity<any, any, any, any>;

const { data } = entity.parse({ Item: rawItem });

[tywalch]

Thanks for putting these together! Luckily, only the "null to remove property" would be breaking 👍

I'll have to find time to respond more completely but here is one thought I had reading the above:

I know it's possible to mock go, but it doesn't provide enough information about the operation currently being executed
The (untyped) go() method on Entity does have the method name, parameters provided, and the execution config. Is that the method you meant?

[zirkelc]

Yes, I meant the go() method and you're right it actually does have the operation name as a parameters 😄 I think what I wanted to express was this: suppose I have a saveUser API endpoint which creates or updates an user. First it has to read if the user exists and then updates it if it exists, or creates a new user if it doesn't. So I have three operations: get, create, update (or patch).

Thsi would result in multiple test cases like:

• should create a new user
• shold update an existing user
• should throw an error if user to be updated does not exist
• ...

When I write unit test for this, I would need to mock go like this (pseudo):

jest.spyOn(Entity.prototype, 'go').mockImplementation((method) => { 
  if(method === "get") return { /* ...*/ };
  if(method === "create") return  { /* ...*/ };
  if(method === "update") return  { /* ...*/ };
})

Ideally, what I would like to do is this:

// setup mocks
const getMock = jest.spyOn(Entity.prototype, 'get');
const createMock = jest.spyOn(Entity.prototype, 'create')
const updateMock = jest.spyOn(Entity.prototype, 'update')

test("should create new user", () => 
  getMock.mockResolveValue({ /* ...*/ });
  createMock.mockResolveValue({ /* ...*/ });

  await saveApi({ name: "joe" });

  expect(getMock).toHaveBeenCalledOnce();
  expect(createMock).toHaveBeenCalledOnce();
  expect(updateMock).not.toHaveBeenCalled();
});

test("should throw an error if...", () => 
  getMock.mockRejectedValue(new Error());

  await saveApi({ id:-1, name: "joe" });

  expect(getMock).toHaveBeenCalledOnce();
  expect(createMock).not.toHaveBeenCalled();
  expect(updateMock).not.toHaveBeenCalled();
});

Having dedicated mockable operation functions integrates much nicer with test frameworks like Jest or Vitest. It's not optimal, but gives more flexibility and control on the returned data.

[tywalch]

This might also be a good add: #411

[zirkelc]

Hi @tywalch

coming back to this, there is another feature I'd like to suggest: an option to ignore readOnly attributes when using .set() on an Entity. All my entities have timestamps createdAt and updatedAt which are set automatically:

attributes: {
  createdAt: {
    type: 'string',
    readOnly: true,
    default: () => isoDateTime(),
    set: () => isoDateTime(),
  },
  updatedAt: {
    type: 'string',
    readOnly: true,
    watch: '*',
    set: () => isoDateTime(),
  },
}

My Update APIs which are calling ElectroDB receive a complete entity with these timestamps, because they were returned from another API, and call Entity.patch(key).set(properties) with the complete entity including the timestamp. Currently, it throws because of the read-only timestamp attributes. Ideally, I'd also like to ignore key attributes in the properties, so that I can really just call Entity.patch(entity).set(entity) and ElectroDB will ignore the keys and read-only attributes in the set() function.

[tywalch]

@zirkelc

I agree that this pattern needs tightening up. I can experiment with the typing here to see if I can differentiate between set callbacks that don't return undefined vs ones that do. Ideally, this one wouldn't need to be a breaking change 👍

[iOSonntag]

Can I ask the question why you would remove the limit parameter? I use it almost everywhere is there something wrong about it?

[tywalch]

Can you say more about how you use it? Its implementation was tricky and unintuitive from a user perspective.

[iOSonntag]

As I understand it, the advantage from limit over count was that if a query returned more items than the specified value you will get those extra items as well correctly?

Well most of our paginated views are endless scroll views so more items do not harm, and saves database costs if the user goes on.

Am I making false assumptions here?

[tywalch]

Was your use of limit (as opposed to the absence of an option) in this case to avoid empty reads?

[iOSonntag]

Sorry I did not understand your question, can you specify it a little bit more?

Let me give you an example of a query I use all over the place. Maybe you can tell me that I used it wrong and should use something else here:

const response = await Chat.query.byUserId_updatedAtUserIdContact({
  userId: user.userId,
})
.go({
  cursor: cursor,
  pages: 'all',
  order: 'desc',
  limit: GameConfig.pageSizeChats,
});

Now obviously I wanted to get chats pages. No cursor 1st page. With cursor another page.

Now from the docs I thought that count and limit when combined with pages: 'all' perform the same algorithm in the background, with the only difference that count will cut exactly after the reached value, where as limit might return more items when the algorithm in the background already have more items at hand.

Given the fact that all paginated views are endless scrollviews we do not care to return the exact amount of items requested but want to display what was read from the database.

[tywalch]

Previously the limit option did a few things:

• Added the dynamodb parameter property Limit to the outbound query request
• Would cause ElectroDB to re-query for results until the number of values was at least reached when paired with pages: 'all'.

(Background you might know, but others often don't) DynamoDB is weird because it doesn't offer a mechanism to retrieve a specific number of items returned. The nature of how Filter Expressions work can result in "empty" reads. DynamoDB queries 1 MB at a time and then applies filters afterward. In the case of an empty receive, there could still be more results to retrieve, but none were found in that 1 MB page.

Now from the docs I thought that count and limit when combined with pages: 'all' perform the same algorithm in the background, with the only difference that count will cut exactly after the reached value, where as limit might return more items when the algorithm in the background already have more items at hand.

This is sort of true, though limit also applied the dynamodb Limit parameter, which often resulted in more requests. The count option does not apply a dynamodb Limit parameter.

From here, there are two big takeaways:

1. In your case, you're not applying additional filters and don't care about how many results you get back. My suggestion would be to avoid the use of limit and count. I would also suggest not using pages: 'all' but you could use pages: 2 or some other number if you want to return more than one page worth.
2. I think there is room for one of two potential new options: atleast (numeric) or seek (boolean). The atleast option would behave similarly to limit but would not add a Limit parameter to DynamoDB. The seek option would repeatedly query until at least one item is returned; this would solve the empty return challenge. Both could be bound by the pages option to mitigate runaway queries.

[iOSonntag]

Thanks for clarification! I might want to update a lot of queries after I have read your information.

Your suggestion might be a great fit (havn't thought hard about it but seems reasonable). Still - let me explain why I went the pages: 'all' + limit route after all. This might help in evaluating the need for the suggested options.

Because ElectroDB does so much for you and sometimes you forget about the underlying DynamoDB actions - it is sometimes hard to tell if the data will be filtered out by the query itself or the afterwards applied DynamoDB FilterExpression.

In a simple query like mine from above its quit obvious that no FilterExpression is applied (except for version check) but sometimes you end up being uncertain if a FilterExpression is applied and how much impact it has on the result percentage. Using my strategy I was certain that every request will return at least that amount of data.

Now to be clear - I knew the 1MB data retrival constraint (but I totally forgot about that in the project) - and thought to minimize cost I applied reasonable pages sizes on my own LOL.

Now after I have read your information it would make sense for me to discard limit and count all together except for 2 special cases. That are:

1
Queries where you expect the FilterExpression to filter out 1MB of data resulting into nearly empty result sets with next cursor in it.

That is because the way my infinite scroll view widgets work is that they would required a user scroll interaction before fetching the next page. (Obviously that could be fixed either client or server side but I would love to let ElectroDB handle this case)

2
Queries where I do not care that I get the rest of the data for free from DynamoDB but want to query just a few (e.g. 10 items) and get a valid next cursor which then can be used to get the next items.

This makes sense in a scenario where you post process those items and want to keep that costs down. For example you populate the resulting items with data from another data source.

Given your Information and my 2 special cases - I would assume now that the first one could be solved with the atleast feature and the second could be solved with the already existent count feature, correct?

[iOSonntag]

And btw you mentioned that the limit feature was hard to implement - if you rather not want to maintain / recreate that feature, I think nobody would be here to blame that

[iOSonntag]

Just one last thought on this: if you think of the atleast feature being the same as the limit feature but without applying the Limit value - does it also means it is exactly the same as the count feature just without hard cutting the items?

If the count feature stays online regardless of the discontinued limit feature, wouldn't it be easy to activate the atleast feature? Don't get me wrong here, nothing is ever easy - but that seems doable I think

[iOSonntag]

Wait a minute: 1MB is something different than 4kb RCU measurement. Meaning that limiting the query does indeed reduce overall costs correctly? What would you recommend in order for reducing costs? If one page parameter is applied with no limit or count, does it mean it automatically fetches an entire 1 MB of data?

[tywalch]

I'm sorry for not getting back to you sooner! Here are a few responses:

Now after I have read your information it would make sense for me to discard limit and count all together except for 2 special cases.

Yes, because you are narrowing your queries complete with indexes, this should work just fine 👍 The exception here is if you're using a clustered index (more info below).

1. Queries where you expect the FilterExpression to filter out 1MB of data resulting into nearly empty result sets with next cursor in it.

Yes, that is one scenario. Another scenario is when you use clustered indexes. In single table design, sometimes other entities can be returned by a query request; ElectroDB will filter these out for you before returning a response, but that's another example of getting an "empty" response.

It is best always to expect that DynamoDB will not give you back any items, but if it returns a cursor, there is still more to query.

Given your Information and my 2 special cases - I would assume now that the first one could be solved with the atleast feature and the second could be solved with the already existent count feature, correct?

Your first example could and should still use count. Using count only becomes less performant if the number you specify is consistently smaller than your query return size. This would be less performant because the number of items returned across the wire by DynamoDB directly correlates to speed. In that case, I would suggest using limit.

a refresher for future readers:
When you use count, ElectroDB will paginate for you until that number of records has been retrieved. For example, if you use { count: 50 }, ElectroDB will query until it has received >= 50. If DynamoDB returns 20 items on each request, ElectroDB would send three requests and receive back 60 items before stopping. ElectroDB would trim the "extra" 10 items and return a cursor that acts as though it only retrieved 50. This type of work is very involved for users to do themselves when their API requires a specific number of items on each query.

Wait a minute: 1MB is something different than 4kb RCU measurement. Meaning that limiting the query does indeed reduce overall costs correctly? What would you recommend in order for reducing costs? If one page parameter is applied with no limit or count, does it mean it automatically fetches an entire 1 MB of data?

DynamoDB changes you for items evaluated on their side. Non-index filters are applied after retrieval so they will never reduce cost. Using limit can reduce cost, but it won't always reduce cost. You're right to bring up RCUs: You are paying for a minimum 1 RCU whether or not you receive 1kb or 4kb. If your limit query returns only 2kb of data, you still pay for 4kb. Worse, turning it into two queries means you'd be paying for 8kb but only receiving 4kb.