Update to AnVIL schema v4 (#4617) #4741

nadove-ucsc · 2022-11-15T04:14:50Z

Connected issues: #4617

Checklist

Author

Target branch is develop
Name of PR branch matches issues/<GitHub handle of author>/<issue#>-<slug>
PR title references all connected issues
PR title matches¹ that of a connected issue _{or comment in PR explains why they're different}
For each connected issue, there is at least one commit whose title references that issue
PR is connected to all connected issues via Zenhub
PR description links to connected issues
Added partial label to PR _{or this PR completely resolves all connected issues}

¹ when the issue title describes a problem, the corresponding PR title is Fix: followed by the issue title

Author (reindex)

Added r tag to commit title _{or this PR does not require reindexing}
Added reindex label to PR _{or this PR does not require reindexing}

Author (chains)

This PR is blocked by previous PR in the chain _{or this PR is not chained to another PR}
Added base label to the blocking PR _{or this PR is not chained to another PR}
Added chained label to this PR _{or this PR is not chained to another PR}

Author (upgrading)

Documented upgrading of deployments in UPGRADING.rst _{or this PR does not require upgrading}
Added u tag to commit title _{or this PR does not require upgrading}
Added upgrade label to PR _{or this PR does not require upgrading}

Author (operator tasks)

Added checklist items for additional operator tasks _{or this PR does not require additional tasks}

Author (hotfixes)

Added F tag to main commit title _{or this PR does not include permanent fix for a temporary hotfix}
Reverted the temporary hotfixes for any connected issues _{or the prod branch has no temporary hotfixes for any connected issues}

Author (requirements, before every review)

Ran make requirements_update _{or this PR does not touch requirements*.txt, common.mk, Makefile and Dockerfile}
Added R tag to commit title _{or this PR does not touch requirements*.txt}
Added reqs label to PR _{or this PR does not touch requirements*.txt}

Author (rebasing, integration test)

make integration_test passes in personal deployment _{or this PR does not touch functionality that could break the IT}
Rebased PR branch on develop, squashed old fixups

Peer reviewer (after requesting changes)

Uncheck the Author (requirements) and Author (rebasing, integration test)
checklists.

Peer reviewer (after approval)

Ticket is in Review requested column
Requested review from primary reviewer
Assigned PR to primary reviewer

Primary reviewer (after requesting changes)

Uncheck the Author (requirements) and Author (rebasing, integration test)
checklists. Update the N reviews label.

Primary reviewer (after approval)

Actually approved the PR
Labeled connected issues as demo or no demo
Commented on connected issues about demo expectations _{or all connected issues are labeled no demo}
Decided if PR can be labeled no sandbox
PR title is appropriate as title of merge commit
N reviews label is accurate
Moved ticket to Approved column
Assigned PR to current operator

Operator (before pushing merge the commit)

Operator (after pushing the merge commit)

¹ When pushing the merge commit is skipped due to the PR being
labelled no sandbox, the next build triggered by a PR whose merge commit is
pushed determines this checklist item.

Operator (reindex)

Deleted unreferenced indices in dev _{or this PR does not remove catalogs or otherwise causes unreferenced indices}
Deleted unreferenced indices in anvildev _{or this PR does not remove catalogs or otherwise causes unreferenced indices}
Started reindex in dev _{or this PR does not require reindexing}
Started reindex in anvildev _{or this PR does not require reindexing}
Checked for and triaged indexing failures in dev _{or this PR does not require reindexing}
Checked for and triaged indexing failures in anvildev _{or this PR does not require reindexing}
Emptied fail queues in dev deployment _{or this PR does not require reindexing}
Emptied fail queues in anvildev deployment _{or this PR does not require reindexing}

Operator

Unassigned PR

Shorthand for review comments

L line is too long
W line wrapping is wrong
Q bad quotes
F other formatting problem

coveralls · 2022-11-15T04:33:56Z

Coverage decreased (-0.02%) to 84.21% when pulling 2196473 on issues/noah-aviel-dove/4617-update-anvil-schema-v4 into c0328ec on develop.

codecov · 2022-11-15T04:36:34Z

Codecov Report

Merging #4741 (c0328ec) into develop (c0328ec) will not change coverage.
The diff coverage is n/a.

❗ Current head c0328ec differs from pull request most recent head 2196473. Consider uploading reports for the commit 2196473 to get more accurate results

@@           Coverage Diff            @@
##           develop    #4741   +/-   ##
========================================
  Coverage    83.74%   83.74%           
========================================
  Files          140      140           
  Lines        17069    17069           
========================================
  Hits         14295    14295           
  Misses        2774     2774

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

achave11-ucsc

LGTM 👌🏽

hannes-ucsc

I am confused about the files names and provenance of the cans.

This comment sounds wrong to me:

azul/scripts/recan_bundle_tdr.py

Line 358 in 64ae91b

Load a canned bundle from DCP/1 and write *.manifest.tdr and *.metadata.tdr

The deletion of .manifest.json and .metadata.json should be an early, separate commit if those files are truly redundant.

The .result and .results in the file names are confusing, very similar but different semantics. I am also confused as to what purpose the .tables file has.

And there's a smell with

azul/scripts/recan_bundle_tdr.py

Line 77 in 64ae91b

def file_paths(parent_dir: str,

I think the code to create it is longer than a literal of the result, even though there is no advantage to making it dynamic since the references are very static.

Please request a PL slot for this.

hannes-ucsc · 2022-12-02T19:21:44Z

scripts/can_bundle.py

@@ -59,8 +67,12 @@ def main(argv):
    parser.add_argument('--output-dir', '-O',
                        default=os.path.join(config.project_root, 'test', 'indexer', 'data'),
                        help='The path to the output directory (default: %(default)s).')
+    parser.add_argument('--redaction-key', '-K',
+                        help='Provide a key to redact personally senstive information from the output files')


Suggested change

help='Provide a key to redact personally senstive information from the output files')

help='Provide a key to redact confidential or sensitive information from the output files')

Please address https://github.com/DataBiosphere/azul/pull/4741/files#r1038456379

scripts/can_bundle.py

hannes-ucsc · 2022-12-02T19:25:13Z

scripts/can_bundle.py

+        o = struct.unpack('>Q', hashlib.sha1(key + str(o).encode()).digest()[:8])[0]
+        o = (o & 0xFFFFFFFFFFFF) + 42000000000000000
+    elif isinstance(o, list):
+        o[:] = sorted(redact_json(item, key) for item in o)


A comment should explain the need for sorting.

hannes-ucsc · 2022-12-02T19:28:47Z

scripts/can_bundle.py

+            redact_json(entity, key)
+
+
+def redact_json(o: AnyMutableJSON, key: bytes) -> AnyMutableJSON:


Either return or mutate in place, never both. If a mix is needed for recursion, a wrapper should hide that. I personally find pure functions easier to write and reason about. I only resort non-pure for performance reasons, or when that is easier to understand, for example, when a surgical change is made. This is more of a transformation so I would write a pure function.

FYI @achave11

hannes-ucsc · 2022-12-06T18:57:26Z

scripts/can_bundle.py

@@ -59,8 +67,12 @@ def main(argv):
    parser.add_argument('--output-dir', '-O',
                        default=os.path.join(config.project_root, 'test', 'indexer', 'data'),
                        help='The path to the output directory (default: %(default)s).')
+    parser.add_argument('--redaction-key', '-K',
+                        help='Provide a key to redact confidential or senstive information from the output files')


Suggested change

help='Provide a key to redact confidential or senstive information from the output files')

help='Provide a key to redact confidential or sensitive information from the output files')

nadove-ucsc added the reindex:dev [process] PR requires reindexing dev label Nov 15, 2022

nadove-ucsc force-pushed the issues/noah-aviel-dove/4617-update-anvil-schema-v4 branch from 607c8fe to e951e22 Compare November 15, 2022 04:15

github-actions bot added the orange [process] Done by the Azul team label Nov 15, 2022

hannes-ucsc force-pushed the issues/noah-aviel-dove/4617-update-anvil-schema-v4 branch from e951e22 to f7a50b3 Compare November 17, 2022 01:56

nadove-ucsc force-pushed the issues/noah-aviel-dove/4617-update-anvil-schema-v4 branch 6 times, most recently from aae2da1 to 275cf4b Compare November 23, 2022 22:28

nadove-ucsc changed the base branch from develop to issues/noah-aviel-dove/4761-anvil-broken-file-links November 23, 2022 22:28

nadove-ucsc added the chained [process] PR needs to based of develop before merging label Nov 23, 2022

nadove-ucsc force-pushed the issues/noah-aviel-dove/4761-anvil-broken-file-links branch from d7b4f27 to 6e6b140 Compare November 29, 2022 00:47

nadove-ucsc force-pushed the issues/noah-aviel-dove/4617-update-anvil-schema-v4 branch from 275cf4b to 18a0979 Compare November 29, 2022 23:45

hannes-ucsc mentioned this pull request Nov 30, 2022

AnVIL indexer doesn't follow downstream links from files to files #4761

Closed

14 tasks

dsotirho-ucsc force-pushed the issues/noah-aviel-dove/4761-anvil-broken-file-links branch from b4fdf1b to 68777aa Compare November 30, 2022 23:53

nadove-ucsc force-pushed the issues/noah-aviel-dove/4617-update-anvil-schema-v4 branch from 18a0979 to 2b1fa2f Compare December 1, 2022 00:01

dsotirho-ucsc changed the base branch from issues/noah-aviel-dove/4761-anvil-broken-file-links to develop December 1, 2022 01:11

dsotirho-ucsc removed the chained [process] PR needs to based of develop before merging label Dec 1, 2022

nadove-ucsc assigned achave11-ucsc Dec 1, 2022

nadove-ucsc requested a review from achave11-ucsc December 1, 2022 01:28

achave11-ucsc approved these changes Dec 1, 2022

View reviewed changes

achave11-ucsc removed their assignment Dec 1, 2022

achave11-ucsc requested a review from hannes-ucsc December 1, 2022 07:21

achave11-ucsc assigned hannes-ucsc Dec 1, 2022

hannes-ucsc requested changes Dec 5, 2022

View reviewed changes

hannes-ucsc added the 1 review [process] Lead requested changes once label Dec 5, 2022

hannes-ucsc removed their assignment Dec 5, 2022

nadove-ucsc requested a review from hannes-ucsc December 6, 2022 01:30

nadove-ucsc assigned hannes-ucsc Dec 6, 2022

hannes-ucsc requested changes Dec 6, 2022

View reviewed changes

hannes-ucsc added 3 reviews [process] Lead requested changes thrice and removed 2 reviews [process] Lead requested changes twice labels Dec 6, 2022

hannes-ucsc removed their assignment Dec 6, 2022

nadove-ucsc force-pushed the issues/noah-aviel-dove/4617-update-anvil-schema-v4 branch 2 times, most recently from 88db91b to 5485050 Compare December 6, 2022 20:25

nadove-ucsc requested a review from hannes-ucsc December 6, 2022 21:00

nadove-ucsc assigned hannes-ucsc Dec 6, 2022

hannes-ucsc approved these changes Dec 7, 2022

View reviewed changes

hannes-ucsc assigned achave11-ucsc and unassigned hannes-ucsc Dec 7, 2022

nadove-ucsc added 11 commits December 7, 2022 10:38

Remove unused canned files

fa4b0f5

Order AnVIL fields consistently with schema document

a700221

[r] Update AnVIL snapshots (#4617)

b695430

Absorb renaming of AnVIL tables (#4617)

e10ffca

Absorb renaming of AnVIL fields (#4617)

9be7c0f

Index all AnVIL v4 schema columns (#4617)

6734467

Remove assay_category

cad6a0d

Use file_name column for downloads

f0e3c70

Index variantcallingactivity entities (#4617)

b018e05

Add redaction support to can_bundle script

ad48032

Replace canned bundle for AnVIL schema v4

2196473

achave11-ucsc force-pushed the issues/noah-aviel-dove/4617-update-anvil-schema-v4 branch from 88db91b to 2196473 Compare December 7, 2022 18:39

achave11-ucsc added the sandbox [process] Resolution is being verified in sandbox deployment label Dec 7, 2022

achave11-ucsc merged commit 05c743d into develop Dec 7, 2022

achave11-ucsc mentioned this pull request Dec 7, 2022

RuntimeError in MP upload tests #4678

Closed

achave11-ucsc deleted the issues/noah-aviel-dove/4617-update-anvil-schema-v4 branch December 8, 2022 00:41

achave11-ucsc removed their assignment Dec 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to AnVIL schema v4 (#4617) #4741

Update to AnVIL schema v4 (#4617) #4741

nadove-ucsc commented Nov 15, 2022 •

edited by achave11-ucsc

Loading

coveralls commented Nov 15, 2022 •

edited

Loading

codecov bot commented Nov 15, 2022 •

edited

Loading

achave11-ucsc left a comment

hannes-ucsc left a comment •

edited

Loading

hannes-ucsc Dec 2, 2022

hannes-ucsc Dec 6, 2022

hannes-ucsc Dec 2, 2022

hannes-ucsc Dec 2, 2022

hannes-ucsc Dec 6, 2022

	help='Provide a key to redact personally senstive information from the output files')
	help='Provide a key to redact confidential or sensitive information from the output files')

		redact_json(entity, key)


		def redact_json(o: AnyMutableJSON, key: bytes) -> AnyMutableJSON:

Update to AnVIL schema v4 (#4617) #4741

Update to AnVIL schema v4 (#4617) #4741

Conversation

nadove-ucsc commented Nov 15, 2022 • edited by achave11-ucsc Loading

Checklist

Author

Author (reindex)

Author (chains)

Author (upgrading)

Author (operator tasks)

Author (hotfixes)

Author (requirements, before every review)

Author (rebasing, integration test)

Peer reviewer (after requesting changes)

Peer reviewer (after approval)

Primary reviewer (after requesting changes)

Primary reviewer (after approval)

Operator (before pushing merge the commit)

Operator (after pushing the merge commit)

Operator (reindex)

Operator

Shorthand for review comments

coveralls commented Nov 15, 2022 • edited Loading

codecov bot commented Nov 15, 2022 • edited Loading

Codecov Report

achave11-ucsc left a comment

Choose a reason for hiding this comment

hannes-ucsc left a comment • edited Loading

Choose a reason for hiding this comment

hannes-ucsc Dec 2, 2022

Choose a reason for hiding this comment

hannes-ucsc Dec 6, 2022

Choose a reason for hiding this comment

hannes-ucsc Dec 2, 2022

Choose a reason for hiding this comment

hannes-ucsc Dec 2, 2022

Choose a reason for hiding this comment

hannes-ucsc Dec 6, 2022

Choose a reason for hiding this comment

nadove-ucsc commented Nov 15, 2022 •

edited by achave11-ucsc

Loading

coveralls commented Nov 15, 2022 •

edited

Loading

codecov bot commented Nov 15, 2022 •

edited

Loading

hannes-ucsc left a comment •

edited

Loading