Deterministic drift computation for Spark

[JanPalasek]

Description

Currently, every time I run a drift computation on Spark, it returns a different result. This is undesirable, since this makes the drift computation not replicable.

Suggestions

I would like to be able to compute drift deterministically, so if I run it on the same data it always returns the same result.

Example

I would assume that this code always returns the same drift.

schema = StructType([StructField("age", IntegerType(), False)])

total_rows = 100000
rng = np.random.default_rng(42)

reference = spark.createDataFrame([Row(age=rng.normal(30, 10)) for _ in range(total_rows)], schema, verifySchema=True)
target = spark.createDataFrame([Row(age=rng.normal(33, 10)) for _ in range(total_rows)], schema, verifySchema=True)

reference_profile_view = collect_dataset_profile_view(
        reference_table,
        schema=DeclarativeSchema(STANDARD_RESOLVER, default_config=MetricConfig(kll_k_large=512)),
)
target_profile_view= collect_dataset_profile_view(
        current_table,
        schema=DeclarativeSchema(STANDARD_RESOLVER, default_config=MetricConfig(kll_k_large=512)),
)

drift = calculate_drift_scores(
        target_view=target_profile_view,
        reference_view=reference_profile_view,
        with_thresholds=True,
        drift_map=kwargs,
)