fix(tests): Fix false negative cram tests

[corneliusroemer]

Description of proposed changes

• Fix Missing escape of square brackets causes diff_jsons.py in some cram tests to be false negative #1665 by properly escaping square brackets
• Add a warning to diff_jsons.py when exclude-regex-path is used incorrectly like in issue 1665 to prevent this issue from reoccurring silently
• Broaden default exclude-paths to now be ["root['generated_by']", "root['meta']['updated']"] instead of ["root['generated_by']['version']"]. This allows simplification of a lot of invocations of diff_jsons.py and in turn reduces chance of errors similar to issue 1665. Anything within generated_by can be ignored by cram tests, meta.updated is only used in export, but no harm excluding by default as it isn't used in any other tests. If there's opposition, happy to revert that change.
• Make the tests pass that should have failed but weren't failing due to issue 1665, by updating the test files. @jameshadfield or @huddlej should check that the behaviour is as intended, but the changes seems small and reasonable so I'm not too worried. Update: @jameshadfield confirmed that the tests didn't mask any unintended behavior.

Checklist

• Automated checks pass
• Check if you need to add a changelog message
• Check if you need to add tests
• Check if you need to update docs

[victorlin]

+1 for using --exclude-paths over --exclude-regex-paths (7c99d29) and for the added defaults (21d2227, bd8d892).

I haven't into validity of the fixed false negatives, but the fixes to the diff script make them seem reasonable.

[victorlin]

Should this include " as well? I'm assuming root['generated_by'] and root["generated_by"] are both valid.

[corneliusroemer]

I think that the path that deep diff uses to match against uses a single quote not double quote. Hence excluding only single quote. Does that make sense?

[victorlin]

nit

Adding or [] (e596b50) doesn't seem useful:

• We don't run mypy/pyright on this file
• A better solution would be to set default=[] on add_argument()

[corneliusroemer]

Just because we don't run mypy doesn't mean that we shouldn't prevent type errors from happening? The error actually happens irrespective of running mypy. It's not mypy erroring it's a real runtime error.

Why is changing the defaults better?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.29%. Comparing base (29188d4) to head (27d18b4).
Report is 4 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1666      +/-   ##
==========================================
+ Coverage   72.26%   72.29%   +0.02%     
==========================================
  Files          79       79              
  Lines        8272     8276       +4     
  Branches     1691     1691              
==========================================
+ Hits         5978     5983       +5     
+ Misses       2009     2008       -1     
  Partials      285      285              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[corneliusroemer]

This might be worth an internal changelog entry