You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The connector partitions the graph to allow Spark to read it in parallel. But Spark does not know anything about the partitioning. Say the connector partitions the graph by predicate and uid range, Spark would not know that and repartition / shuffle the read data if it wanted to join on partition or uid. If Spark would know the exact partitioning scheme, it could avoid un-needed shuffle steps.
Check to what extend Spark allows data sources to tell it about its partitioning.
The text was updated successfully, but these errors were encountered:
The connector partitions the graph to allow Spark to read it in parallel. But Spark does not know anything about the partitioning. Say the connector partitions the graph by predicate and uid range, Spark would not know that and repartition / shuffle the read data if it wanted to join on partition or uid. If Spark would know the exact partitioning scheme, it could avoid un-needed shuffle steps.
Check to what extend Spark allows data sources to tell it about its partitioning.
The text was updated successfully, but these errors were encountered: