Fix: AnVIL indexer doesn't follow downstream links from files to files (#4761) #4765

nadove-ucsc · 2022-11-23T21:40:40Z

Connected issues: #4761

Checklist

Author

Target branch is develop
Name of PR branch matches issues/<GitHub handle of author>/<issue#>-<slug>
PR title references all connected issues
PR title matches¹ that of a connected issue _{or comment in PR explains why they're different}
For each connected issue, there is at least one commit whose title references that issue
PR is connected to all connected issues via Zenhub
PR description links to connected issues
Added partial label to PR _{or this PR completely resolves all connected issues}

¹ when the issue title describes a problem, the corresponding PR title is Fix: followed by the issue title

Author (reindex)

Added r tag to commit title _{or this PR does not require reindexing}
Added reindex label to PR _{or this PR does not require reindexing}

Author (chains)

This PR is blocked by previous PR in the chain _{or this PR is not chained to another PR}
Added base label to the blocking PR _{or this PR is not chained to another PR}
Added chained label to this PR _{or this PR is not chained to another PR}

Author (upgrading)

Documented upgrading of deployments in UPGRADING.rst _{or this PR does not require upgrading}
Added u tag to commit title _{or this PR does not require upgrading}
Added upgrade label to PR _{or this PR does not require upgrading}

Author (operator tasks)

Added checklist items for additional operator tasks _{or this PR does not require additional tasks}

Author (hotfixes)

Added F tag to main commit title _{or this PR does not include permanent fix for a temporary hotfix}
Reverted the temporary hotfixes for any connected issues _{or the prod branch has no temporary hotfixes for any connected issues}

Author (requirements, before every review)

Ran make requirements_update _{or this PR does not touch requirements*.txt, common.mk, Makefile and Dockerfile}
Added R tag to commit title _{or this PR does not touch requirements*.txt}
Added reqs label to PR _{or this PR does not touch requirements*.txt}

Author (rebasing, integration test)

make integration_test passes in personal deployment _{or this PR does not touch functionality that could break the IT}
Rebased PR branch on develop, squashed old fixups

Peer reviewer (after requesting changes)

Uncheck the Author (requirements) and Author (rebasing, integration test)
checklists.

Peer reviewer (after approval)

Ticket is in Review requested column
Requested review from primary reviewer
Assigned PR to primary reviewer

Primary reviewer (after requesting changes)

Uncheck the Author (requirements) and Author (rebasing, integration test)
checklists. Update the N reviews label.

Primary reviewer (after approval)

Actually approved the PR
Labeled connected issues as demo or no demo
Commented on connected issues about demo expectations _{or all connected issues are labeled no demo}
Decided if PR can be labeled no sandbox
PR title is appropriate as title of merge commit
N reviews label is accurate
Moved ticket to Approved column
Assigned PR to current operator

Operator (before pushing merge the commit)

Operator (after pushing the merge commit)

¹ When pushing the merge commit is skipped due to the PR being
labelled no sandbox, the next build triggered by a PR whose merge commit is
pushed determines this checklist item.

Operator (reindex)

Deleted unreferenced indices in dev _{or this PR does not remove catalogs or otherwise causes unreferenced indices}
Deleted unreferenced indices in anvildev _{or this PR does not remove catalogs or otherwise causes unreferenced indices}
Started reindex in dev _{or this PR does not require reindexing}
Started reindex in anvildev _{or this PR does not require reindexing}
Checked for and triaged indexing failures in dev _{or this PR does not require reindexing}
Checked for and triaged indexing failures in anvildev _{or this PR does not require reindexing}
Emptied fail queues in dev deployment _{or this PR does not require reindexing}
Emptied fail queues in anvildev deployment _{or this PR does not require reindexing}

Operator

Unassigned PR

Shorthand for review comments

L line is too long
W line wrapping is wrong
Q bad quotes
F other formatting problem

coveralls · 2022-11-23T22:02:12Z

Coverage remained the same at 84.514% when pulling 68777aa on issues/noah-aviel-dove/4761-anvil-broken-file-links into 55336ab on develop.

codecov · 2022-11-23T22:04:08Z

Codecov Report

Merging #4765 (b4fdf1b) into develop (55336ab) will increase coverage by 0.01%.
The diff coverage is 0.00%.

❗ Current head b4fdf1b differs from pull request most recent head 68777aa. Consider uploading reports for the commit 68777aa to get more accurate results

@@             Coverage Diff             @@
##           develop    #4765      +/-   ##
===========================================
+ Coverage    84.02%   84.04%   +0.01%     
===========================================
  Files          140      140              
  Lines        16938    16932       -6     
===========================================
- Hits         14233    14231       -2     
+ Misses        2705     2701       -4

Impacted Files	Coverage Δ
src/azul/plugins/metadata/anvil/__init__.py	`77.35% <ø> (ø)`
...c/azul/plugins/metadata/anvil/indexer/transform.py	`99.44% <ø> (ø)`
...rc/azul/plugins/metadata/anvil/service/response.py	`38.33% <ø> (ø)`
src/azul/plugins/repository/tdr_anvil/__init__.py	`32.81% <0.00%> (ø)`
src/azul/terraform.py	`74.81% <0.00%> (+1.76%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

dsotirho-ucsc

Approved.

src/azul/plugins/repository/tdr_anvil/__init__.py

hannes-ucsc · 2022-11-28T23:07:56Z

src/azul/plugins/repository/tdr_anvil/__init__.py

-                    ala.alignmentactivity_id,
-                    'alignmentactivity',
+                    ala.alignmentactivity_id AS activity_id,
+                    'alignmentactivity' AS activity_table,


This seems to be an unrelated change, maybe a fix for a bug that was shadowed by #4761? If so, that should go into a separate commit.

hannes-ucsc · 2022-11-28T23:08:04Z

src/azul/plugins/repository/tdr_anvil/__init__.py

-                        activity=KeyReference(key=row['actvity_id'], entity_type=row['activity_table']))
+                        activity=KeyReference(key=row['activity_id'], entity_type=row['activity_table']))


This reverts commit 6859944.

…files (#4761)

hannes-ucsc · 2022-12-05T19:10:17Z

@danielsotirhos, when you are operator, you must not unassign yourself with open checklist items.

dsotirho-ucsc · 2022-12-05T19:25:23Z

I previously failed to mark the checkbox Build passes on GitLab anvildev for the merge commit.
Marking the checkbox now, proof of build passing is here: https://gitlab.anvil.gi.ucsc.edu/ucsc/azul/-/pipelines/2498

nadove-ucsc added base [process] Another PR needs to be rebased before merging this one reindex:dev [process] PR requires reindexing dev labels Nov 23, 2022

github-actions bot added the orange [process] Done by the Azul team label Nov 23, 2022

nadove-ucsc force-pushed the issues/noah-aviel-dove/4761-anvil-broken-file-links branch from d7b4f27 to 04bf2d5 Compare November 23, 2022 21:57

nadove-ucsc requested a review from dsotirho-ucsc November 23, 2022 22:26

nadove-ucsc assigned dsotirho-ucsc Nov 23, 2022

dsotirho-ucsc previously approved these changes Nov 24, 2022

View reviewed changes

dsotirho-ucsc requested a review from hannes-ucsc November 24, 2022 01:28

dsotirho-ucsc assigned hannes-ucsc and unassigned dsotirho-ucsc Nov 24, 2022

hannes-ucsc requested changes Nov 28, 2022

View reviewed changes

hannes-ucsc added the 1 review [process] Lead requested changes once label Nov 28, 2022

hannes-ucsc removed their assignment Nov 28, 2022

nadove-ucsc dismissed dsotirho-ucsc’s stale review via 6e6b140 November 29, 2022 00:47

nadove-ucsc force-pushed the issues/noah-aviel-dove/4761-anvil-broken-file-links branch 2 times, most recently from 1025d91 to 35d3b27 Compare November 29, 2022 05:11

nadove-ucsc requested a review from hannes-ucsc November 29, 2022 05:59

nadove-ucsc assigned hannes-ucsc Nov 29, 2022

hannes-ucsc approved these changes Nov 30, 2022

View reviewed changes

hannes-ucsc assigned dsotirho-ucsc and unassigned hannes-ucsc Nov 30, 2022

nadove-ucsc added 7 commits November 30, 2022 15:52

Clean up removed field from AnVIL response

ed23b1a

Remove redundant column in AnVIL indexer

4bf25b3

Revert "Work around string file sizes in CMG snapshots"

4c2ff67

This reverts commit 6859944.

Fix incorrect field name in AnVIL metadata mapping

6847085

Refactor bailouts

c7bf7da

Fix broken query

25864f6

[r] Fix: AnVIL indexer doesn't follow downstream links from files to …

68777aa

…files (#4761)

dsotirho-ucsc force-pushed the issues/noah-aviel-dove/4761-anvil-broken-file-links branch from b4fdf1b to 68777aa Compare November 30, 2022 23:53

dsotirho-ucsc added the sandbox [process] Resolution is being verified in sandbox deployment label Nov 30, 2022

dsotirho-ucsc approved these changes Dec 1, 2022

View reviewed changes

dsotirho-ucsc merged commit 68c0e09 into develop Dec 1, 2022

dsotirho-ucsc removed the base [process] Another PR needs to be rebased before merging this one label Dec 1, 2022

dsotirho-ucsc removed their assignment Dec 1, 2022

hannes-ucsc assigned achave11-ucsc Dec 5, 2022

achave11-ucsc deleted the issues/noah-aviel-dove/4761-anvil-broken-file-links branch December 5, 2022 22:52

achave11-ucsc removed their assignment Dec 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix: AnVIL indexer doesn't follow downstream links from files to files (#4761) #4765

Fix: AnVIL indexer doesn't follow downstream links from files to files (#4761) #4765

nadove-ucsc commented Nov 23, 2022 •

edited by achave11-ucsc

Loading

coveralls commented Nov 23, 2022 •

edited

Loading

codecov bot commented Nov 23, 2022 •

edited

Loading

dsotirho-ucsc left a comment

hannes-ucsc Nov 28, 2022 •

edited

Loading

hannes-ucsc Nov 28, 2022

hannes-ucsc commented Dec 5, 2022

dsotirho-ucsc commented Dec 5, 2022

		activity=KeyReference(key=row['actvity_id'], entity_type=row['activity_table']))
		activity=KeyReference(key=row['activity_id'], entity_type=row['activity_table']))

Fix: AnVIL indexer doesn't follow downstream links from files to files (#4761) #4765

Fix: AnVIL indexer doesn't follow downstream links from files to files (#4761) #4765

Conversation

nadove-ucsc commented Nov 23, 2022 • edited by achave11-ucsc Loading

Checklist

Author

Author (reindex)

Author (chains)

Author (upgrading)

Author (operator tasks)

Author (hotfixes)

Author (requirements, before every review)

Author (rebasing, integration test)

Peer reviewer (after requesting changes)

Peer reviewer (after approval)

Primary reviewer (after requesting changes)

Primary reviewer (after approval)

Operator (before pushing merge the commit)

Operator (after pushing the merge commit)

Operator (reindex)

Operator

Shorthand for review comments

coveralls commented Nov 23, 2022 • edited Loading

codecov bot commented Nov 23, 2022 • edited Loading

Codecov Report

dsotirho-ucsc left a comment

Choose a reason for hiding this comment

hannes-ucsc Nov 28, 2022 • edited Loading

Choose a reason for hiding this comment

hannes-ucsc Nov 28, 2022

Choose a reason for hiding this comment

hannes-ucsc commented Dec 5, 2022

dsotirho-ucsc commented Dec 5, 2022

nadove-ucsc commented Nov 23, 2022 •

edited by achave11-ucsc

Loading

coveralls commented Nov 23, 2022 •

edited

Loading

codecov bot commented Nov 23, 2022 •

edited

Loading

hannes-ucsc Nov 28, 2022 •

edited

Loading