Optimize VCF import

[Arkanosis]

Import in regovar/core/managers/imports/vcf_manager.py is slow. Therefore:

• find a test VCF to import;
• create an empty pg database;
• import the VCF into the database;
• profile the import;
• either improve the existing code or rewrite it (possibly ditching pysam, sqlalchemy or python in the process, if they happen to prevent some optimization).

@Oodnadatta @ikit : do you please have a relevant, annotated VCF for me, that is slow enough to import? The slower the better (I can shorten it as needed if that's really too slow for me). Thanks!

[ikit]

@Arkanosis : You will find several annotated VCF on Brownie, in the /var/regovar/files directory

[ikit]

The new importer should be wrap in a pipeline to become the new official vcf importer : https://github.com/REGOVAR-Pipelines/VCFImporter

[Arkanosis]

Ok great! :)

[Arkanosis]

Which means by the way that we really have no reason to use python more than anything else, right?

[ikit]

yes

[ikit]

And in addition to this task, we should have a reflexion on the DB schema, I have the intuition that we can remove the "variant id". I thinks that we need it only in the "working table" but if we can avoid to use it ("insert or update" query) it at the step of the import, we will be able to increase a lot performance.

see : https://github.com/REGOVAR/Regovar/blob/master/regovar/core/managers/imports/vcf_manager.py#L569