Set a new descriptive schema on mutations if none exists #383
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
New code avoids decoding the mutation metadata if there is none there. This reduces the time to 310 seconds for the metadata setting on chr2q. I think this is probably a fair compromise: it's always going to take time to validate 12 million JSON entries rather than just dumping raw bytes. |
|
Aha - it's the validation that takes time: the encoding is quick. I've made a change to only bother validating if there is existing data, and that brings it down to 95 secs |
|
I'm just going to merge this now. Is that OK @nspope ? Do you want to look over it beforehand? |
And emit warning if mutation metadata cannot be set
Also allow overwriting, and save node time schema using this
hyanwong
force-pushed
the
set-mutation-schema
branch
from
1873687
to
3e8cc6b
Compare
hyanwong
force-pushed
the
set-mutation-schema
branch
from
d1ade6f
to
f730460
Compare
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
And emit warning if mutation metadata cannot be set. This also means that existing mutation metadata is kept. However, since we now validate the metadata, it increases the time taken. E.g. dating the unified genealogy chr 2q (12462233 mutations) the metadata setting goes from 84 seconds to 403 seconds on my laptop (for comparison, the 10 EP steps take ~ 7 mins).
There might be a way to do this faster without validation if no node metadata already exists, I suppose.