Using validation results in the normalisation process

[rhiaro]

A NormalisedEvent:

• should not contain any fields with empty or null values.
• should have all fields that are not in the OA modelling spec or prefixed with beta: moved to an invalidAttributes object.

The best way to detect both of these is from the results of validating the object. We either need to:

• be able to guarantee the validation is run before the normalisation pipes, so we can access the results from the database, or
• run the validator from inside the pipes; which might be a significant performance hit.

[rhiaro]

Normalisations

[nickevansuk]

Just a note that beta: attributes sometimes give useful information. A decision was made in modelling spec development to encourage the use of beta: over further iterations of the modelling spec, so hence many useful properties remain in beta, and beta is now quite extensive as a result.

beta properties can be readily validated against https://openactive.io/ns-beta/beta.jsonld if that's helpful

[nickevansuk]

To make this relevant to the present COVID-19 situation, most of the properties that describe virtual events are in fact in beta, for the same reason

[rhiaro]

Thanks @nickevansuk. Current behaviour is to pass beta properties through and republish them unchanged, so nothing should be lost there.

[rhiaro]

Data cleaning pipe