CMDCT-3960: creating new dynamo table for measures with PKs that can be queried instead of scan all

[angelaco11]

Description

The current dynamo measures table has a limitation in its primary key structure. It currently is {state}{year}{coreSet}{measure}. The limitation here is the fact that the measure is tacked on to the primary key which makes each primary key unique. Therefore, when we want to do a query to get all the measures for a state, year and coreset, we cannot do a query on the primary key (since it contains each individual measure) and must do a scanAll which is not performant and is causing db threshold alarms.

The changes here are to write a script that creates a new measures table that is rekeyed. The new primary key is {state}{year}{coreSet} and the range/sort key is the measure name. this allows us to query for all measures for a state, year and coreset since all of those measures will have the same primary key (e.g. all measures from AL, 2024 and CCSM will have the primary key AL2024CCSM. The sort/range key will be the measure so that single measures can be retrieved by combining the primary key and the range key. This will make querying the db much more performant and we will not have to use scanAll.

I also updated all calls to query the measures table to point to the new table with the correct keys.

Related ticket(s)

CMDCT-3960

---

How to test

https://d2flx34i0vw50w.cloudfront.net/

1. Log in as any user
2. Click on any coreset and make sure that the measures list loads properly

Notes

---

Pre-review checklist

• I have added thorough tests, if necessary
• I have updated relevant documentation, if necessary
• I have performed a self-review of my code
• I have manually tested this PR in the deployed cloud environment

---

Pre-merge checklist

Review

• Design: This work has been reviewed and approved by design, if necessary
• Product: This work has been reviewed and approved by product owner, if necessary

Security

If either of the following are true, notify the team's ISSO (Information System Security Officer).

• These changes are significant enough to require an update to the SIA.
• These changes are significant enough to require a penetration test.

---

convert to a different template: test → val | val → prod

[ajaitasaini]

thank you for the thorough description + all the context provided -- made it so easy to understand why we're making these changes! testing via cloudfront worked as expected and the code makes sense / nothing major is sticking out to me

[benmartin-coforma]

Looks good to me!

[benmartin-coforma]

@ailZhou Note that we will need to perform the migration in this PR immediately when it merges into each environment, or we risk data loss.

[codeclimate]

Code Climate has analyzed commit b01c160 and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (90% is the threshold).

This pull request will bring the total coverage in the repository to 84.1% (-0.1% change).

View more on Code Climate.

[rocio-desantiago]

Code looks good 👍
Tested with deploy link: Measures showing up for each year as expected ✅

[ailZhou]

lgtm! I ran it on deploy and i read through the code to the best of my abilities and nothing seemed weird to me.

Thank you for taking on this huge task of migrating the database and increasing QMR's performance!