Composable Labels

[willr3]

Problem:

The dataset -> Label -> [Change, Experiment, Report, etc] workflow is detrimental

• Too many concepts for users to learn
• Duplicated implementation details increase development work
• Does not produce consistent output for different configurations
• Limits workflow to two or three steps of data manipulation
• Conceals Dataset transformation recombination from users

Proposal:

Change label extractors so that they can process input from either the initial run or from the output of other labels. This gives users the ability to define a pipeline of data processing that is as simple or complex as they need to handle their data and decreases the model complexity inside Horreum.

Labels would have 3 forms of extractors:

runDataExtractor : $.json.path.in.data

This is the extractor we already have and it runs a jsonpath against the run data uploaded by the user.

metadataExtractor : {metadata}:$.json.path.in.metadata

This is an extractor that accesses information in the Horreum run table that is outside the run data uploaded by the user. There is not a well defined way for view_components to access the run’s ID. That is currently handled with custom additional parameters to the render function. This would allow an extractor with {id} to put the run’s id value into a standard method of data access for view_components.
This type of extractor would also codify access to any run metadata using json path extraction.

labelValueExtractor : labelName:$.json.path.in.data

This is the new composite extractor that runs against the output of another Label. If defining the extractor in a single string it is labelName with an optional colon separator followed by an optional jsonpath.

labelName

The simplest would be just the labelName which would pass the label_value directly to the current label’s combination function.

labelName:$.json.path.in.data

This combines the labelName with a jsonpath. It extracts the value found by the jsonpath from the output of the named label (label_value)

labelName[*]

The [*] suffix means we want horreum to iterate over the values from the named label and run the combination function N times where N is the number of values from the named label. This is easy to do when the label_value is an array. If the label_value is an object do we assume the user wants to iterate over entries? {key: [key], value: [value]}?

The resulting N values from the N executions of the combination function would be stored as an array in the label_values table but we would need to store an indicator that the array is actually separate values. If this label is passed as input to another label then that other label would automatically trigger N times because from the user perspective the label has N values from the N executions of the previous label.

Note: Consider adding the [*] at the end of the extractor jsonpath to indicate it is an iterating extractor. This allows iteration in one Label rather than having to first extract a value in LabelA then iterate on LabelA’s label_value in LabelB. The downside to this notation is that the suffix is not universal to all jsonpath dialects so it would not be compatible with all possible persistence options (e.g. jsopath-ng and jq).

Label States

Labels will be in different states depending on the type and number of extractors they use.

Iterating Extractor with non-iterating extractor(s)

If a label has an iterating labelValueExtractor and other non-iterating extractors then the user needs to tell us if they want the values from the other extractors passed into the combination function every time it runs or only the first time it runs.
I propose we do this with a configuration option (because there are only 2 options) similar to how Grafana allows configuring transforms from a list of selectable transforms. We would create some info-graphics that illustrate how the data is handled and naturally have copy text available to describe it.

Multiple Iterating Extractors

If a label has more than one iterating labelValueExtractor then the user has to inform us if they want N x N executions of the combination function or Nn where Nn is the maximum length of the iterating functions array.

Maximum Length

If they choose the maximum length option do we ask the user what they want passed into the combination function for the shorter iterating label_value?. Do they want no value (null) or do they want Horreum to repeat previous iterations of the value? I would suspect no value and perhaps that is our default behavior but we should consider users may have other needs.

There is a possible condition where the label also has non-itetering extractors that produce a single value. We have to ask what the user wants to do for those values when the iterating labels run the combination function on the second and subsequent iterations.

N x N Combinations

The users could want to run the combination function for every permutation of the iterating extractors. This seems unlikely but we should support this option.

Initial Implementation

This change to label extractors will allow us to replace several redundant concepts within Horreum. Our initial implementation will replace Datasets with Labels where Tests are producing more than one dataset from a run (approximately 10% of tests).

Dataset

The existing transformation_extractor and transformation_function will become the extractor and combination function for a new label (labelA). This new label will be an intermediary label with an array output because existing tranformation_functions output an array to indicate multiple datasets.
Horreum would then need a labelB with an iterating labelValueExtractor on labelA to indicate to subsequent labels (labelC…) that the label_value should be treated as distinct entries and labelC should run on each entry in the label_value for labelB.
This change means label_values must store a state indicator to tell subsequent labels that the value contains multiple distinct entries in an array.

UI Changes

Changing the data mode will naturally change how we represent data to the user.

Test Dataset

Datasets would no longer exist. Users would instead create “schema views'' which are tables that show a row for each label_value from the test that target a specific schema based on the label.target_schema. The table would include fixed columns (e.g. the run id where the label_value originated) as well as columns based on any label_values derived from the label_value with the target schema.

Horreum would need to either track the label dependency graph (it is a graph not a tree) to know which labels come from label_values that match the target schema or perform that graph walk when the user wants to add a column to the table view. The graph walk would be based on the label and label_extractor tables, not label_values.

Implementation Requirements

Extractor Encoding

Users should be able to easily change extractors and this means changing them between types (runData, labelValue, metadata). I think a good first implementation is to allow users to define the extractors as a sting and for Horreum to identify the type from the string. Changing existing extractors would potentially delete and create a new extractor if it changes type. A shared table would help change between encodings but isn’t strictly necessary.

Label Loop Detection

Changing an extractor or Label name could create a dependency loop. It is technically possible for a dependency loop to resolve but the risk of infinite looping means we should detect and prevent loops. I propose we limit labels to uni-directional dependencies with branch and merge options but no loops.

Label Value Tracking

Iterating Label Indicator

We plan to store values from iterating Labels in a single entry in label_values by encoding them as an array. We need to differentiate between arrays from iterating Labels and arrays from non-iterating Labels because Labels that depend on iterating Labels will need to create multiple values.

Label Source Indicator

Changing the UI to creating tables from label_values based on the label target_schema means we need to be able to identify which label_value are derived from those target_schema label_values. Non-iterating extractors store the label source in the extractor but iterating extractors create a different source index for each subsequent label.
If LabelX depends on N iterating labelValueExtractors that extract from Label_alpha, Label_beta… then Horreum will have to store a reference to the index in the label_values for Label_alpha and Label_beta for each index in the label_values for LabelX. There could be N references for each entry in the label_values for Label_X where N is the number of extractors for Label_X (if all extractors are iterating labelValueExtractors).

[willr3]

Summary

• 3 extractors (Jsonpath, Metadata, Labelvalue)
• Tests own Labels (partly for PoC simplicity)
• Users create Extractors from strings
• Looped Label references fail validation (Label.isCircular)
• Label dependency detection for label value creation and updates
• Hibernate Validation for Label and Extractor Invariants
• Proper Hibernate Model dependency

Design Discussion

• Change to a single Extractor class with invariant validation?
• Referential integrity for entities?
• Additional cases in the sql to extract values?
• @EmbeddedId or multiple @Id annotations?
  Hibernate supports multiple @Id but an @EmbeddedId is JPA compliant

Extractors

JsonpathExtractor - what we already have in Horreum

$.* 

RunMetadataExtractor - access columns other than data

{metadata}:$.*

LabelValueExtractor - extract from previous label values

labelName
labelName[]
labelName:$.*
labelName[]:$.*

Extractor names are not unique for a label so that they can override using last in first out. This allows users to create multiple extractors to support changes to the run data format where known data changes location.

Label Value Creation

The PoC creates all Label Values in Horreum not in the database (no sql inserts). Labels are sorted using Kahn Directed Acyclic Graph sort to ensure label values are available when a LabelValueExtractor needs them. The PoC still creates a json object from all the extracted values like current Horreum. There are two key sql operations:

io.hyperfoil.tools.exp.horreum.svc.LabelService#calculateExtractedValuesWithIterated

with m as (select e.name, e.dtype, e.jsonpath, e.foreach, e.column_name, lv.data as lv_data, lv.iterated as lv_iterated, r.data as run_data, r.metadata as run_metadata from extractor e left join label_values lv on e.target_id = lv.label_id, run r  where e.parent_id = :label_id and (lv.run_id = :run_id or lv.run_id is null) and r.id = :run_id),
n as (select m.name, m.dtype, m.jsonpath, m.foreach, m.lv_iterated ,m.lv_data, (case
    when m.dtype = 'JsonpathExtractor' and m.jsonpath is not null then jsonb_path_query_array(m.run_data,m.jsonpath::::jsonpath)
    when m.dtype = 'RunMetadataExtractor' and m.jsonpath is not null and m.column_name = 'metadata' then jsonb_path_query_array(m.run_metadata,m.jsonpath::::jsonpath)
    when m.dtype = 'LabelValueExtractor' and m.jsonpath is not null and m.jsonpath != '' and m.lv_iterated then extract_path_array(m.lv_data,m.jsonpath::::jsonpath)
    when m.dtype = 'LabelValueExtractor' and m.jsonpath is not null and m.jsonpath != '' then jsonb_path_query_array(m.lv_data,m.jsonpath::::jsonpath)
    when m.dtype = 'LabelValueExtractor' and (m.jsonpath is null or m.jsonpath = '') then to_jsonb(ARRAY[m.lv_data])
    else '[]'::::jsonb end) as found from m)
select n.name as name,(case when jsonb_array_length(n.found) > 1 then n.found else n.found->0 end) as data, n.lv_iterated as lv_iterated from n 

which then uses a custom sql function

CREATE OR REPLACE FUNCTION extract_path_array(_arr jsonb, _path jsonpath)
  RETURNS jsonb
  LANGUAGE sql STABLE PARALLEL SAFE AS
'
    with bag as (SELECT jsonb_path_query_array(elem,_path) as data FROM jsonb_array_elements(_arr) elem)
    select jsonb_agg((case when jsonb_array_length(data) > 1 then data else data->0 end)) from bag;
';; 

Label Value Interaction

Horreum currently allows a run to create multiple datasets and thus multiple rows in the dataset table. The PoC replaces this with target_schemas on Labels so that the Label Values are claimed to adhere to a given schema and users can fetch all label_values that implement that schema

io.hyperfoil.tools.exp.horreum.svc.LabelService#getBySchema

or get a map of sub-label values for each label value that implements a schema. Getting the sub-label maps is the equivalent of getting the ExtractedLabelValues for datasets.

io.hyperfoil.tools.exp.horreum.svc.LabelService#labelValues(java.lang.String, long, java.util.List<java.lang.String>, java.util.List<java.lang.String>)

[johnaohara]

@willr3 do you have a link to the POC repo?

[willr3]

The Poc is here but I plan to delete the repo once we finalize any designs and merge any useful parts.
https://github.com/willr3/Poc_ComposableLabel

[willr3]

I think the name target_schma implies Horreum will treat the resulting label values as though they adhere to a schema in Horreum. Users could potentially want Horreum to apply the labels from the named schema.

[willr3]

User Workflows

How does a user replace the Datasets view where a single run upload could have multiple rows?

Users create Label(s) with a common target_schema (in the PoC it is a string). The target_schema tells Horreum that the output of multiple labels are simliar objects and that the user wants to track sub values from that targeted value. The LabelService has a labelValues(String Schema...) endpoint that allows Horreum to retrieve all the label values (in a map by label name) that are derived from each instance of the label value that adheres to the target_schema. That is quite technical. Think of the target_schema as the name "Dataset" and labelValues("Dataset") would fetch the ExtractedLabelValueMap for each Dataset. This allows the Horreum UI to create views based on target_schemas and know the labels that are applicable / available for custom columns.

If a user has two labels with a common target_schema how do they extract label values without duplicating labels?

if Label_A and Label_B both have a target_schema="Dataset" and the user wants to define a set of common labels for all instances of Dataset without duplicating the labels they should add a Label_AB that merges the two arrays {a,b}=>[...a,...b] Whatever label pipeline lead to Label_A and Label_B could be merged at that point because they are meant to be the same (they have the same target_schema). The target_schema would then be defined on Label_AB instead of Label_A Label_B so that Horreum tracks the derived labels from that new set of label values.

How does the user handle if the run data changes over time?

This depends on the type and complexity of the change. If Label_A used an extractor named UUID to extract from $.run.metadata.UUID but the new schema moved it to $.ids.UUID then the current PoC implementation supports defining a second extractor named UUID with the $.ids.UUID jsonpath. The PoC uses the last value to match (and maintains the order of extractors). That means if the new jsonpath did not exist before but starts to exist afterwards then the PoC will pick the new jsonpath when available.
If both jsonpaths always existed and there is some separate criteria to decide when to switch between $.ids and $.runs then the user should create a new label UUID that extracts both paths and uses a reducer (currently javascript) plus an additional run data to determine which value to accept. The original Label_A would change the extractor from $.run.metadata.UUID to UUID which means it would change from a JsonpathExtractor to a LabelValueExtractor and store the value based on the logic in the reducer.

[lampajr]

Hey,

I think I got the overall approach and how it is intended to work (and to be implemented).

What I still do not have fully clear is this sentence

We plan to store values from iterating Labels in a single entry in label_values by encoding them as an array. We need to differentiate between arrays from iterating Labels and arrays from non-iterating Labels because Labels that depend on iterating Labels will need to create multiple values.

especially

because Labels that depend on iterating Labels will need to create multiple values.

Why do we need to mark a label to be "iterating" or "non-iterating" for subsequent labels, I mean IIUC the idea is to encode the result of iterating labels as an array (therefore a single label_value with an array as value, right?), given that shouldn't be a responsability of the label that will use the previous one to decide how to manage the previous result (i.e., the input, if that is an array)? In that way it can decide to apply a map over all entries or take the array as single input and produce a single output which can be an array as well.

This way the label should specify how it will treat the source label input, i.e., if the current label itself is a map function/operation over an array of entries that will produce an array of entries or a simple one-to-one function label.

I don't get if I am over-simplifying the approach or if I am missing some use cases that would make this approach not working.

---

Another thing that I think could be useful, at least from my perspective, is a sort of ER diagram showing how data would be mapped into tables' records. Especially for the scenario where we replace the existing dataset with this new model, having a concrete ER diagram showing how the data would like like in the database could be very helpful to have a clear picture of the to be scenario. I'd be very happy to help on this!

[lampajr]

Thanks Will, now it is clearer 🙏

I was not considering that the iterating label extractor could be an intermediate step, and the users who created that, will probably create subsequent labels assuming that the input will be an iterating one. Thus, I see why we should keep track if the label output will be in the iterating form or not

We could tell users we are storing it all in one array and require the user to create Labels that know our implementation detail but I do not think we should allow our implementation choices to change how users conceptualize their data flow.

Yeah I agree on this, we should hide implementation details to the end user.

[willr3]

I've not created or worked with Entity Relationship diagrams but I see mermaid has that format so I'll start reading :)

[lampajr]

yeah, I mentioned ER because it would be easier to switch between ER diagrams and database tables/schemas but whatever you prefer is fine for me (+1 to use mermaid as it will be easier to share and to inject in markdowns).

I think the main point here is trying to "visualize" the relationship between involved entities and how they will change wrt the "as is" model, if you need any help lmk.

[willr3]

erDiagram
    Test ||--|{ Label : defines
    Label ||--|{ Extractor : defines
    Extractor ||--o| Label: "depends on"
    LabelValue |{--|| Run: "depends on"
    Label ||--|{ LabelValue: "creates"
    Test ||--|{ Run: "applies to"

Loading

[willr3]

The above diagram does not include the LabelValuePointer table that stores the relationship from entries in LabelValue to other entires in LabelValues. I thought creating multiple LabelValue's from a single Label would mean Horreum no longer needed to track the LabelValuePointer but one LabelValue can have a relationship to multiple other LabelValues (if the label has multiple LabelValueExtractors). Creating multiple LabelValues entries from a single Label would allow us to remove the childIndex and targetIndex columns from LabelValueExtractor.

[johnaohara]

@willr3 Thanks for writing this up. I agree this would a be great step forward to simplifying the current concepts and reduce the cognitive load of users

1. There are different syntax for different types, e.g;

$.json.path.in.data
{metadata}:$.json.path.in.metadata
labelName:$.json.path.in.data

in order to make it simpler for users, should we have a consistent syntax:

{run}:$.json.path.in.data
{metadata}:$.json.path.in.metadata
{labelName}:$.json.path.in.data

if we were to do this, we need reserved words for labelName, i.e. can not be run / metadata

2. labelName is sematically equivalent to labelName:$. or ({labelName}:$. if we were to adopt the above syntax).

If we want to reduce the cognitive load, do we want to support / encourage the shortcuts? or make it explicit what users want, this would simplify the code slightly as well

3. labelName[*] requires functionality in horreum to iterate over an array and apply a combination function to each value in the array, it then needs to store a flag to denote that the result is an array. Could this also be handled in the combination function, i.e. the user that is writing the function knows that input is an object or an array, and can chose to handle an array anyway they like, i.e. processing the array and returning an object/scalar, or processing the array and returning an array. This later would be equivalent, and we would not have to handle this special case in Horreum code? This would also simplify the combinations of iterating vs non-iterating extractors etc.

The current proposal simplifies the reducer function.

3. Datasets - the original idea behind datasets was to allow splitting of run data into multiple json docs that are not directly comparable. This was then used for change detection, experiments etc. Rather than storing a state indicator in label_value, can we not define a label as an input for change detection, if the label is an array, treat it as separate non-comparable json objects, if it is an object, treat it as a single result? I think the same for views and everywhere else datasets are currently used.

[willr3]

1. I think there is value in progressive enhancement and in visual differentiation. I also disagree that the syntax is different. The syntax was designed as prefix : jsonpath where prefix can include the iteration indicator [] and both prefix and jsonpath are optional. prefixes wrapped in {} refer to Horreum managed fields and prefixes without {} refer to user managed label names.
   90% of tests in production do not use metadata and do not use the Datasets feature. We should not impose additional overhead on those use cases (e.g. $.json.path.in.data instead of {run}:$.json.path.in.data). The {run} prefix would need to be optional otherwise we add complexity to the common use case to support the uncommon use cases. The current Horreum design forced all users to use and understand Datasets despite it only applying to 10% of tests and I think we should avoid repeating that design.
   This does raise the issue of "reserved words." The current PoC does not limit names for labels but we will need to prevent conflict with whatever we use as the prefix for the supported Horreum fields. I would prefer to keep Horreum fields as the only names that are surrounded in curly brackets so {bizbuzz} would not be a valid label name but bizbuzz or metadata would be fine.
   There is also potential conflict with label names starting with $ (e.g. the cursed $schema). The easiest solution would be disallow $ as the first character. We could alternatively check if the $... is a label name and give that precedence over jsonpath evaluation. The second option would require a two phase parse to building extractors: the first pass collects the list of possible label names and the the second uses that list to detect label name references. We may want to adopt the two phase approach in either case to avoid the temporary entities that are currently created in the PoC.
   I personally like the visual differentiation of {name} versus name as an indication that {name} refers to a Horreum managed attribute and name refers to a user's label name. I would argue changing the string encoding to {metadata} and {name} increases the user cognitive load because there is no longer a clear visual indicator which one is referencing a Label and which one is referencing a Horreum field. Using brackets for both fields and label names would be forcing users to reference another location in the UI to determine where the extractor is finding {metadata} until they build up the knowledge of all the reserved Horreum field names.

I do not think the syntax is without its problems.
The iteration indicator [] is borrowed from jq's jsonpath dialect (postgres uses [*]). It further limits valid label names (cannot end with []). It also lacks affordance. Users familiar with jsonpath will recognize it as the jq method to individually address the entries in an array but having it oustide the jsonpath means we have to document and educate users about the affordance. I considered relying on iteration being defined in the jsonpath (e.g. $[*] or $.path.to.array[*]). I'm still not convinced labelName[] is best but I went with that representation to avoid overloading the existing jsonpath dialect with implicit interpretations. An unknowing user could write $[*] and think that is how to get all the array entries (which is valid) and not mean to treat them as independent entries for subsequent labels. The labelName[] approach has only one valid interpretation (thus explicit invocation).

2. labelName is not semantically equivalent to labelName:$. because $. is not valid jsonpath for all valid json encoded values. The problem is $. only works for an object, it will not work for an array or a scalar, both of which could be stored in a label_value.

3. labelName[*] can be accomplished with use javascript (reducer code) but that causes problems for Horreum. If the user wants to create dataset equivalent independent objects in Horreum we need to be able to track label values that are associated with those independent objects. We can do this if Horreum is responsible for the iteration but if the user's javascript controls the iteration then Horreum will not be able to perform the association. There is nothing in the proposal that stops a user from having their existing transformation function that produces an array and having a label that identifies each entry in that array as an independent object (e.g. with an extractor transformed[] and a target_schema). If Horreum does not offer the transformed[] functionality then when the user wants to extract say $.name from each entry in the transformed array Horreum will not be able to identify which entry in the transformed array created which $.name output without enforcing some additional representational constraint on the users.
   The current Dataset transformation is an example of this representational constraint / abstraction leak. Transformation functions create arrays where Horreum will treat each entry in the array as a Dataset. effectively, Transformation functions create an array that is "owned" by Horreum and the entries inside the array are the data the user owns. Horreum internally stores Datasets in an array so this is effectively the internal design of Horreum bleeding into the user requirements.

4. I think we should avoid implicit assumptions about user intent when data is "a certain shape." Change detection should be a scope (default is run but it could be a target_schema), a set of fingerprinting labels (controls), an observation label (variable), and algorithm specific tuning that is applicable to the variable. Our current "change detection" algorithms work with numeric scalars but there are algorithms that support distributions, non-parametric observations, etc. If we make assumptions about user intent based on the content of the variable we make it difficult for Horreum to adapt to additional algorithims in the future.
   I do, however, support giving the user sane defaults or UI feedback when they are configuring their change detection.

splitting of run data into multiple json docs that are not directly comparable

The current PoC supports separated json docs with target_schema (btw I do not think we should use that name when we merge). If the separate json docs are the same shape then they can use the same target_schema but multiple target_schema would also work. Extracted label values can be retrieved and grouped by ancestor target_schema. This allows users to treat an instance of a target_schema as an independent json object the same way we treat.