Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update to AnVIL schema v4 #4617

Closed
4 of 6 tasks
hannes-ucsc opened this issue Oct 19, 2022 · 7 comments
Closed
4 of 6 tasks

Update to AnVIL schema v4 #4617

hannes-ucsc opened this issue Oct 19, 2022 · 7 comments
Assignees
@nadove-ucsc
Labels
code [subject] Production code demo [process] To be demonstrated at the end of the sprint demoed [process] Successfully demonstrated to team enh [type] New feature or request orange [process] Done by the Azul team

Comments

@hannes-ucsc
Copy link
Member

hannes-ucsc commented Oct 19, 2022 •
edited by nadove-ucsc
Loading

https://github.com/broadinstitute/anvil_tdr_ingest/pull/2/files

  • Switch the v3 snapshots in the corpus to their updated v4 counterparts
  • Absorb renaming of fields
  • Absorb renaming of tables
  • Ensure that we index all tables
  • Ensure that we follow all foreign keys in the schema
  • Ensure that we index all columns in the schema

I don't think we currently follow the FK from_biosample.diagnosis_id_to_diagnosis.diagnosis_id that's being removed in v4. So nothing to do on that front. But see #4618.

Absorbing the renamed tables doesn't affect our internal naming of entity types i.e. biosample stays biosample even though the table is called anvil_biosample.
@hannes-ucsc hannes-ucsc added the orange [process] Done by the Azul team label Oct 19, 2022
@melainalegaspi
Copy link

@noah-aviel-dove received review request, waiting for review to complete before triaging.

@melainalegaspi melainalegaspi added enh [type] New feature or request indexer [subject] The indexer part of Azul code [subject] Production code and removed indexer [subject] The indexer part of Azul labels Oct 21, 2022
@melainalegaspi
Copy link

@theathorn to monitor spreadsheet for updated snapshots.

@theathorn theathorn mentioned this issue Oct 25, 2022
Closed
@hannes-ucsc
Copy link
Member Author

hannes-ucsc commented Nov 4, 2022 •
edited
Loading

Current v4 snapshots will be retracted and replacements with more file connections (partially addressing #4654) will be released.

https://ucsc-gi.slack.com/archives/C03TPJS54DC/p1667564548110159

@theathorn
Copy link

Slack thread re. updated snapshots.

@theathorn
Copy link

Rows 29-38 contain the 10 updated snapshots.

@theathorn theathorn assigned nadove-ucsc and unassigned theathorn Nov 8, 2022
@nadove-ucsc nadove-ucsc mentioned this issue Nov 15, 2022
Merged
75 tasks
hannes-ucsc pushed a commit that referenced this issue Nov 17, 2022
@nadove-ucsc
[r] Update AnVIL snapshots (#4617)
56ce0e7
hannes-ucsc pushed a commit that referenced this issue Nov 17, 2022
@nadove-ucsc
Absorb renaming of AnVIL tables (#4617)
f7a50b3
nadove-ucsc added a commit that referenced this issue Nov 22, 2022
@nadove-ucsc
[r] Update AnVIL snapshots (#4617)
60f6229
nadove-ucsc added a commit that referenced this issue Nov 22, 2022
@nadove-ucsc
Absorb renaming of AnVIL tables (#4617)
274baae
nadove-ucsc added a commit that referenced this issue Nov 22, 2022
@nadove-ucsc
Absorb renaming of AnVIL fields (#4617)
04ec6b0
nadove-ucsc added a commit that referenced this issue Nov 22, 2022
@nadove-ucsc
Index all AnVIL v4 schema columns (#4617)
4be1111
nadove-ucsc added a commit that referenced this issue Nov 22, 2022
@nadove-ucsc
Index all AnVIL v4 schema columns (#4617)
8bed6f8
nadove-ucsc added a commit that referenced this issue Nov 22, 2022
@nadove-ucsc
Absorb renaming of AnVIL fields (#4617)
487fb5b
nadove-ucsc added a commit that referenced this issue Nov 22, 2022
@nadove-ucsc
Index all AnVIL v4 schema columns (#4617)
926ba74
nadove-ucsc added a commit that referenced this issue Nov 23, 2022
@nadove-ucsc
[r] Update AnVIL snapshots (#4617)
d8ff74c
nadove-ucsc added a commit that referenced this issue Nov 23, 2022
@nadove-ucsc
Absorb renaming of AnVIL tables (#4617)
96a97be
nadove-ucsc added a commit that referenced this issue Nov 29, 2022
@nadove-ucsc
Absorb renaming of AnVIL fields (#4617)
2557823
nadove-ucsc added a commit that referenced this issue Nov 29, 2022
@nadove-ucsc
Index all AnVIL v4 schema columns (#4617)
81d7395
nadove-ucsc added a commit that referenced this issue Nov 29, 2022
@nadove-ucsc
Index variantcallingactivity entities (#4617)
2669987
@hannes-ucsc hannes-ucsc mentioned this issue Nov 30, 2022
Closed
14 tasks
nadove-ucsc added a commit that referenced this issue Dec 1, 2022
@nadove-ucsc
[r] Update AnVIL snapshots (#4617)
5eac3f5
nadove-ucsc added a commit that referenced this issue Dec 1, 2022
@nadove-ucsc
Absorb renaming of AnVIL tables (#4617)
b29014a
nadove-ucsc added a commit that referenced this issue Dec 1, 2022
@nadove-ucsc
Absorb renaming of AnVIL fields (#4617)
e181948
nadove-ucsc added a commit that referenced this issue Dec 1, 2022
@nadove-ucsc
Index all AnVIL v4 schema columns (#4617)
0f2dc69
nadove-ucsc added a commit that referenced this issue Dec 1, 2022
@nadove-ucsc
Index variantcallingactivity entities (#4617)
e804288
nadove-ucsc added a commit that referenced this issue Dec 5, 2022
@nadove-ucsc
[r] Update AnVIL snapshots (#4617)
b686858
nadove-ucsc added a commit that referenced this issue Dec 5, 2022
@nadove-ucsc
Absorb renaming of AnVIL tables (#4617)
aa8816e
nadove-ucsc added a commit that referenced this issue Dec 5, 2022
@nadove-ucsc
Absorb renaming of AnVIL fields (#4617)
e931c56
nadove-ucsc added a commit that referenced this issue Dec 5, 2022
@nadove-ucsc
Index all AnVIL v4 schema columns (#4617)
db71808
nadove-ucsc added a commit that referenced this issue Dec 5, 2022
@nadove-ucsc
Index variantcallingactivity entities (#4617)
72a422c
nadove-ucsc added a commit that referenced this issue Dec 6, 2022
@nadove-ucsc
[r] Update AnVIL snapshots (#4617)
6957e78
nadove-ucsc added a commit that referenced this issue Dec 6, 2022
@nadove-ucsc
Absorb renaming of AnVIL tables (#4617)
acf9e20
nadove-ucsc added a commit that referenced this issue Dec 6, 2022
@nadove-ucsc
Absorb renaming of AnVIL fields (#4617)
8157dd3
nadove-ucsc added a commit that referenced this issue Dec 6, 2022
@nadove-ucsc
Index all AnVIL v4 schema columns (#4617)
a49cd09
nadove-ucsc added a commit that referenced this issue Dec 6, 2022
@nadove-ucsc
Index variantcallingactivity entities (#4617)
26237c0
@hannes-ucsc
Copy link
Member Author

hannes-ucsc commented Dec 7, 2022 •
edited
Loading

For demo, show that each snapshot is represented on the /index/files endpoint. Name the tables and foreign keys that are not indexed/followed and discuss if we need additional tickets for them.

@hannes-ucsc hannes-ucsc added the demo [process] To be demonstrated at the end of the sprint label Dec 7, 2022
achave11-ucsc pushed a commit that referenced this issue Dec 7, 2022
@nadove-ucsc @achave11-ucsc
[r] Update AnVIL snapshots (#4617)
b695430
achave11-ucsc pushed a commit that referenced this issue Dec 7, 2022
@nadove-ucsc @achave11-ucsc
Absorb renaming of AnVIL tables (#4617)
e10ffca
achave11-ucsc pushed a commit that referenced this issue Dec 7, 2022
@nadove-ucsc @achave11-ucsc
Absorb renaming of AnVIL fields (#4617)
9be7c0f
achave11-ucsc pushed a commit that referenced this issue Dec 7, 2022
@nadove-ucsc @achave11-ucsc
Index all AnVIL v4 schema columns (#4617)
6734467
achave11-ucsc pushed a commit that referenced this issue Dec 7, 2022
@nadove-ucsc @achave11-ucsc
Index variantcallingactivity entities (#4617)
b018e05
achave11-ucsc added a commit that referenced this issue Dec 7, 2022
@achave11-ucsc
[r] Update to AnVIL schema v4 (#4617, PR #4741)
05c743d
@nadove-ucsc
Copy link
Contributor

Follow-ups from demo:
#4827
#4828

@hannes-ucsc hannes-ucsc added the demoed [process] Successfully demonstrated to team label Dec 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nadove-ucsc nadove-ucsc

Labels
code [subject] Production code demo [process] To be demonstrated at the end of the sprint demoed [process] Successfully demonstrated to team enh [type] New feature or request orange [process] Done by the Azul team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@hannes-ucsc @theathorn @melainalegaspi @nadove-ucsc