You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There are a few obvious things we need to change about the way graph specs are processed.
We should have the ability to specify multiple graph specs at once and queue up building graphs from multiple specs at once.
We also need the ability to have sub graph dependencies cross over from one spec to another. For example the Baseline graph is shared by robokopkg and yobokop but currently there's no way to have it in one place and have each reference the same thing.
Right now you can build just one graph from a graph spec but it still checks for latest versions of every source in the spec (for sources that don't have a pinned version). This is bad because when you just want to build one graph in a spec, it's a waste of time to check them all, and a failed version check could disrupt building a graph that doesn't even use that source.
The text was updated successfully, but these errors were encountered:
It might also be nice, but is a way lower priority, to have the ability to reference another graph but say you want to build a that graph without a particular source. For example the rule mining kp is the baseline minus tmkp, but currently there's no way to do that without just making another copy of the spec that's mostly redundant.
Another thing that is simple but would be a nice change, is that currently if the load_manager pipeline fails for a single data source it crashes the entire graph it was part of, but it should just continue to process the rest of the data sources (but not attempt to build the graph) so that when you come back and fix the failure the rest of the work is done
There are a few obvious things we need to change about the way graph specs are processed.
We should have the ability to specify multiple graph specs at once and queue up building graphs from multiple specs at once.
We also need the ability to have sub graph dependencies cross over from one spec to another. For example the Baseline graph is shared by robokopkg and yobokop but currently there's no way to have it in one place and have each reference the same thing.
Right now you can build just one graph from a graph spec but it still checks for latest versions of every source in the spec (for sources that don't have a pinned version). This is bad because when you just want to build one graph in a spec, it's a waste of time to check them all, and a failed version check could disrupt building a graph that doesn't even use that source.
The text was updated successfully, but these errors were encountered: