Filtering out embargoed data

[dosumis]

• We need to add a filtering step so that datasets can be embargoed. In pipeline 1, this step is at the OWL export stage. In p2 it should also be between KB and the integration layer.

• The step is essential to having a complete, functioning pipeline.

• Filtering works by matching a graph pattern, starting from DataSet. Note, graph patterns specifying filters are likely to be close or identical to the ones we need to schema validation. We can base specification on the filtering code used in owl generation.

[dosumis]

This Cypher pattern documents a standard pattern of relationships from a single individual

MATCH (ds:DataSet) where ds.production
WITH ds
MATCH lp=(ds)-[:has_license]-(l:License)
WITH ds, lp
OPTIONAL MATCH pp=(ds)-[:has_reference]->(p:pub)
WITH ds, lp, pp
MATCH ip=(ds)<-[:has_source]-(i:Individual)<-[:depicts]-(ch:Individual)-[:in_register_with]->(tc:Individual)-[:depicts]-(t)
WITH ds, lp, pp, ip, i
MATCH icp=(i)-[:INSTANCEOF|Related|hasDbXref]->(c)
RETURN lp,pp, ip, icp limit 2

TBA - (i)-[]-(i) where bother are in production
(cc)-[]-(Class

In addition - all classes and relationships between them should be loaded.

[dosumis]

Alternative approach - delete entities under embargo after full release

1. Delete all ds:DataSet where ds.production is False
2. Delete all i:Individual where (ds)-[:has_source]-(i:Individual)<-[:depicts]-(ch:Individual) WHERE ds.production is False

[matentzn]

If there is no production flag = true, delete.

[matentzn]

Flag for broken images that should be filtered out.

[matentzn]

2 flags:

• block: true. If block: true, remove the edge/remove the node
• production: true
• staging: true (if production is true, staging is true as well)

We want to be able to run a branch of the pipeline in staging mode.

[matentzn]

blocking on plugin level.

[matentzn]

• Double check this is the list of currently embargoed data:

http://virtualflybrain.org/data/CostaJefferis_v2
http://virtualflybrain.org/data/TrumanWood2018
http://virtualflybrain.org/data/FlyLight2019LateralHorn2019
http://virtualflybrain.org/data/FlyLight2019Strother2017
http://virtualflybrain.org/data/Shih2020
http://virtualflybrain.org/data/FlyLight2019Wolff2018
http://virtualflybrain.org/data/Tsubouchi2017
http://virtualflybrain.org/data/FlyLight2019Namiki2018
http://virtualflybrain.org/data/FlyLight2019Hampel2015

[matentzn]

Manual tests:

Exploring a non-embargoed channel from the Aso2014 dataset:

http://ts.p2.virtualflybrain.org/rdf4j-workbench/repositories/vfb/explore?resource=%3Chttp%3A%2F%2Fvirtualflybrain.org%2Freports%2FVFBc_00103377%3E&limit_explore=100&show-datatypes=show-dataypes

Exploring an embargoes channel from the TrumanWood2018 dataset:
http://ts.p2.virtualflybrain.org/rdf4j-workbench/repositories/vfb/explore?resource=%3Chttp%3A%2F%2Fvirtualflybrain.org%2Freports%2FVFBc_00053706%3E&limit_explore=100&show-datatypes=show-dataypes

Exploring all mentions of the production property:

http://ts.p2.virtualflybrain.org/rdf4j-workbench/repositories/vfb/explore?resource=%3Chttp%3A%2F%2Fn2o.neo%2Fcustom%2Fproduction%3E&limit_explore=100&show-datatypes=show-dataypes

@dosumis this is now ready for your review.