Skip to content
CI

Support diff for Spark Connect (Dataset API) #1179

Sign in to view logs

Support diff for Spark Connect (Dataset API)

Support diff for Spark Connect (Dataset API) #1179

GitHub Actions / Test Results failed Aug 16, 2024 in 0s

21 fail, 9 skipped, 478 pass in 1h 22m 45s

   547 files   -  60     547 suites   - 60   1h 22m 45s ⏱️ - 14m 13s
   508 tests ±  0     478 ✅  -   9    9 💤 +  9   21 ❌ ±  0 
18 719 runs   - 930  18 267 ✅  - 891  158 💤  - 228  294 ❌ +189 

Results for commit 05625da. ± Comparison against earlier commit f423cf4.

Annotations

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_dataframe_diff (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 5s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 5s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 5s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 5s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 5s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 6s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 5s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 6s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 5s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 5s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 5s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 4s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 4s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 5s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_dataframe_diff>

    def test_dataframe_diff(self):
>       diff = self.left_df.diff(self.right_df, 'id').orderBy('id').collect()

python/test/test_diff.py:145: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:586: in diff
    return Differ().diff(self, other, *id_columns)
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ab4f088b0>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_dataframe_diff_with_changes (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_dataframe_diff_with_changes>

    def test_dataframe_diff_with_changes(self):
        options = DiffOptions().with_change_column('changes')
>       diff = self.left_df.diff_with_options(self.right_df, options, 'id').orderBy('id').collect()

python/test/test_diff.py:164: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:625: in diff_with_options
    return Differ(options).diff(self, other, *id_columns)
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ab4f0ea90>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_dataframe_diff_with_default_options (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_dataframe_diff_with_default_options>

    def test_dataframe_diff_with_default_options(self):
>       diff = self.left_df.diff_with_options(self.right_df, DiffOptions(), 'id').orderBy('id').collect()

python/test/test_diff.py:154: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:625: in diff_with_options
    return Differ(options).diff(self, other, *id_columns)
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ab4f08430>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_dataframe_diff_with_diff_mode_column_by_column (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_dataframe_diff_with_diff_mode_column_by_column>

    def test_dataframe_diff_with_diff_mode_column_by_column(self):
        options = DiffOptions().with_diff_mode(DiffMode.ColumnByColumn)
>       diff = self.left_df.diff_with_options(self.right_df, options, 'id').orderBy('id').collect()

python/test/test_diff.py:169: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:625: in diff_with_options
    return Differ(options).diff(self, other, *id_columns)
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ac69fe490>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_dataframe_diff_with_diff_mode_left_side (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_dataframe_diff_with_diff_mode_left_side>

    def test_dataframe_diff_with_diff_mode_left_side(self):
        options = DiffOptions().with_diff_mode(DiffMode.LeftSide)
>       diff = self.left_df.diff_with_options(self.right_df, options, 'id').orderBy('id').collect()

python/test/test_diff.py:179: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:625: in diff_with_options
    return Differ(options).diff(self, other, *id_columns)
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ab41d7040>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_dataframe_diff_with_diff_mode_right_side (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_dataframe_diff_with_diff_mode_right_side>

    def test_dataframe_diff_with_diff_mode_right_side(self):
        options = DiffOptions().with_diff_mode(DiffMode.RightSide)
>       diff = self.left_df.diff_with_options(self.right_df, options, 'id').orderBy('id').collect()

python/test/test_diff.py:184: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:625: in diff_with_options
    return Differ(options).diff(self, other, *id_columns)
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ab41f89d0>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_dataframe_diff_with_diff_mode_side_by_side (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_dataframe_diff_with_diff_mode_side_by_side>

    def test_dataframe_diff_with_diff_mode_side_by_side(self):
        options = DiffOptions().with_diff_mode(DiffMode.SideBySide)
>       diff = self.left_df.diff_with_options(self.right_df, options, 'id').orderBy('id').collect()

python/test/test_diff.py:174: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:625: in diff_with_options
    return Differ(options).diff(self, other, *id_columns)
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ac681e2e0>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_dataframe_diff_with_options (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_dataframe_diff_with_options>

    def test_dataframe_diff_with_options(self):
        options = DiffOptions('d', 'l', 'r', 'i', 'c', 'r', 'n', None)
>       diff = self.left_df.diff_with_options(self.right_df, options, 'id').orderBy('id').collect()

python/test/test_diff.py:159: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:625: in diff_with_options
    return Differ(options).diff(self, other, *id_columns)
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ac69fec70>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_dataframe_diff_with_sparse_mode (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_dataframe_diff_with_sparse_mode>

    def test_dataframe_diff_with_sparse_mode(self):
        options = DiffOptions().with_sparse_mode(True)
>       diff = self.left_df.diff_with_options(self.right_df, options, 'id').orderBy('id').collect()

python/test/test_diff.py:189: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:625: in diff_with_options
    return Differ(options).diff(self, other, *id_columns)
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ac69aad00>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_dataframe_diffwith (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_dataframe_diffwith>

    def test_dataframe_diffwith(self):
>       diff = self.left_df.diffwith(self.right_df, 'id').orderBy('id').collect()

python/test/test_diff.py:149: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:605: in diffwith
    return Differ().diffwith(self, other, *id_columns)
python/gresearch/spark/diff/__init__.py:350: in diffwith
    diff = self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ac69bea60>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_diff_with_epsilon_comparator (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_diff_with_epsilon_comparator>

    def test_diff_with_epsilon_comparator(self):
        # relative inclusive epsilon
        options = DiffOptions() \
            .with_column_name_comparator(DiffComparators.epsilon(0.1).as_relative().as_inclusive(), 'val')
>       diff = self.left_df.diff_with_options(self.right_df, options, 'id').orderBy('id').collect()

python/test/test_diff.py:343: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:625: in diff_with_options
    return Differ(options).diff(self, other, *id_columns)
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ab41d7160>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_differ_diff (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_differ_diff>

    def test_differ_diff(self):
>       diff = Differ().diff(self.left_df, self.right_df, 'id').orderBy('id').collect()

python/test/test_diff.py:193: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ac9b4e730>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_differ_diff_in_diff_mode_column_by_column (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_differ_diff_in_diff_mode_column_by_column>

    def test_differ_diff_in_diff_mode_column_by_column(self):
        options = DiffOptions().with_diff_mode(DiffMode.ColumnByColumn)
>       diff = Differ(options).diff(self.left_df, self.right_df, 'id').orderBy('id').collect()

python/test/test_diff.py:218: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ab4f0ec70>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_differ_diff_in_diff_mode_left_side (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_differ_diff_in_diff_mode_left_side>

    def test_differ_diff_in_diff_mode_left_side(self):
        options = DiffOptions().with_diff_mode(DiffMode.LeftSide)
>       diff = Differ(options).diff(self.left_df, self.right_df, 'id').orderBy('id').collect()

python/test/test_diff.py:228: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ac69b1520>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_differ_diff_in_diff_mode_right_side (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_differ_diff_in_diff_mode_right_side>

    def test_differ_diff_in_diff_mode_right_side(self):
        options = DiffOptions().with_diff_mode(DiffMode.RightSide)
>       diff = Differ(options).diff(self.left_df, self.right_df, 'id').orderBy('id').collect()

python/test/test_diff.py:233: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ab41f8250>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_differ_diff_in_diff_mode_side_by_side (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_differ_diff_in_diff_mode_side_by_side>

    def test_differ_diff_in_diff_mode_side_by_side(self):
        options = DiffOptions().with_diff_mode(DiffMode.SideBySide)
>       diff = Differ(options).diff(self.left_df, self.right_df, 'id').orderBy('id').collect()

python/test/test_diff.py:223: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ac69349a0>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_differ_diff_with_changes (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_differ_diff_with_changes>

    def test_differ_diff_with_changes(self):
        options = DiffOptions().with_change_column('changes')
>       diff = Differ(options).diff(self.left_df, self.right_df, 'id').orderBy('id').collect()

python/test/test_diff.py:213: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ac6998430>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_differ_diff_with_default_options (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_differ_diff_with_default_options>

    def test_differ_diff_with_default_options(self):
        options = DiffOptions()
>       diff = Differ(options).diff(self.left_df, self.right_df, 'id').orderBy('id').collect()

python/test/test_diff.py:203: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ac686a520>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_differ_diff_with_options (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_differ_diff_with_options>

    def test_differ_diff_with_options(self):
        options = DiffOptions('d', 'l', 'r', 'i', 'c', 'r', 'n', None)
>       diff = Differ(options).diff(self.left_df, self.right_df, 'id').orderBy('id').collect()

python/test/test_diff.py:208: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ac6932880>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_differ_diff_with_sparse_mode (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_differ_diff_with_sparse_mode>

    def test_differ_diff_with_sparse_mode(self):
        options = DiffOptions().with_sparse_mode(True)
>       diff = Differ(options).diff(self.left_df, self.right_df, 'id').orderBy('id').collect()

python/test/test_diff.py:238: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:326: in diff
    return self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ac67d7c10>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check warning on line 0 in test.test_diff.DiffTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

All 14 runs failed: test_differ_diffwith (test.test_diff.DiffTest) 

artifacts/Python Test Results (Spark 3.0.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785755.034606091-9982.xml [took 0s]
artifacts/Python Test Results (Spark 3.1.3 Scala 2.12.10 Python 3.8)/test-results/pytest-1723785751.302203469-32407.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785777.859188033-28691.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785787.150269178-20849.xml [took 0s]
artifacts/Python Test Results (Spark 3.2.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785792.937901301-1141.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.10)/test-results/pytest-1723785756.625719308-15728.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.8)/test-results/pytest-1723785746.253537151-14514.xml [took 0s]
artifacts/Python Test Results (Spark 3.3.4 Scala 2.12.15 Python 3.9)/test-results/pytest-1723785741.763594264-25985.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.10)/test-results/pytest-1723785752.792842472-15106.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.8)/test-results/pytest-1723785741.968328620-1172.xml [took 0s]
artifacts/Python Test Results (Spark 3.4.2 Scala 2.12.17 Python 3.9)/test-results/pytest-1723785744.885088192-15810.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.10)/test-results/pytest-1723785742.263449250-27193.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.8)/test-results/pytest-1723785744.631883915-10858.xml [took 0s]
artifacts/Python Test Results (Spark 3.5.1 Scala 2.12.18 Python 3.9)/test-results/pytest-1723785772.326496606-5372.xml [took 0s]
Raw output 
AttributeError: 'function' object has no attribute 'conf'
self = <test.test_diff.DiffTest testMethod=test_differ_diffwith>

    def test_differ_diffwith(self):
>       diff = Differ().diffwith(self.left_df, self.right_df, 'id').orderBy('id').collect()

python/test/test_diff.py:197: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/gresearch/spark/diff/__init__.py:350: in diffwith
    diff = self._do_diff(left, right, list(id_columns), [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gresearch.spark.diff.Differ object at 0x7f3ac6927cd0>
left = DataFrame[id: bigint, val: double, label: string]
right = DataFrame[id: bigint, val: double, label: string], id_columns = ['id']
ignore_columns = []

    def _do_diff(self, left: DataFrame, right: DataFrame, id_columns: List[str], ignore_columns: List[str]) -> DataFrame:
        """
          private def doDiff[T, U](
              left: Dataset[T],
              right: Dataset[U],
              idColumns: Seq[String],
              ignoreColumns: Seq[String] = Seq.empty
          ): DataFrame = {
            checkSchema(left, right, idColumns, ignoreColumns)
    
            val columns = left.columns.diffCaseSensitivity(ignoreColumns).toList
            val pkColumns = if (idColumns.isEmpty) columns else idColumns
            val valueColumns = columns.diffCaseSensitivity(pkColumns)
            val valueStructFields = left.schema.fields.map(f => f.name -> f).toMap
            val valueVolumnsWithComparator = valueColumns.map(c => c -> options.comparatorFor(valueStructFields(c)))
    
            val existsColumnName = distinctPrefixFor(left.columns) + "exists"
            val leftWithExists = left.withColumn(existsColumnName, lit(1))
            val rightWithExists = right.withColumn(existsColumnName, lit(1))
            val joinCondition =
              pkColumns.map(c => leftWithExists(backticks(c)) <=> rightWithExists(backticks(c))).reduce(_ && _)
            val unChanged = valueVolumnsWithComparator
              .map { case (c, cmp) =>
                cmp.equiv(leftWithExists(backticks(c)), rightWithExists(backticks(c)))
              }
              .reduceOption(_ && _)
    
            val changeCondition = not(unChanged.getOrElse(lit(true)))
    
            val diffActionColumn =
              when(leftWithExists(existsColumnName).isNull, lit(options.insertDiffValue))
                .when(rightWithExists(existsColumnName).isNull, lit(options.deleteDiffValue))
                .when(changeCondition, lit(options.changeDiffValue))
                .otherwise(lit(options.nochangeDiffValue))
                .as(options.diffColumn)
    
            val diffColumns = getDiffColumns(pkColumns, valueColumns, left, right, ignoreColumns).map(_._2)
            val changeColumn = getChangeColumn(existsColumnName, valueVolumnsWithComparator, leftWithExists, rightWithExists)
              // turn this column into a sequence of one or none column so we can easily concat it below with diffActionColumn and diffColumns
              .map(Seq(_))
              .getOrElse(Seq.empty[Column])
    
            leftWithExists
              .join(rightWithExists, joinCondition, "fullouter")
              .select((diffActionColumn +: changeColumn) ++ diffColumns: _*)
          }
        """
        self._check_schema(left, right, id_columns, ignore_columns)
    
>       case_sensitive = left.session.conf.get("spark.sql.caseSensitive") == "true"
E       AttributeError: 'function' object has no attribute 'conf'

python/gresearch/spark/diff/__init__.py:430: AttributeError

Check notice on line 0 in .github

See this annotation in the file changed.

@github-actions github-actions / Test Results

9 skipped tests found 

There are 9 skipped tests, see "Raw output" for the full list of skipped tests.
Raw output 
test.test_jvm.PackageTest ‑ test_create_temp_dir
test.test_jvm.PackageTest ‑ test_dotnet_ticks
test.test_jvm.PackageTest ‑ test_get_jvm_connect
test.test_jvm.PackageTest ‑ test_histogram
test.test_jvm.PackageTest ‑ test_install_pip_package
test.test_jvm.PackageTest ‑ test_install_poetry_project
test.test_jvm.PackageTest ‑ test_job_description
test.test_jvm.PackageTest ‑ test_parquet
test.test_jvm.PackageTest ‑ test_with_row_numbers

Check notice on line 0 in .github

See this annotation in the file changed.

@github-actions github-actions / Test Results

508 tests found 

There are 508 tests, see "Raw output" for the full list of tests.
Raw output 
test.test_diff.DiffTest ‑ test_dataframe_diff
test.test_diff.DiffTest ‑ test_dataframe_diff_with_changes
test.test_diff.DiffTest ‑ test_dataframe_diff_with_default_options
test.test_diff.DiffTest ‑ test_dataframe_diff_with_diff_mode_column_by_column
test.test_diff.DiffTest ‑ test_dataframe_diff_with_diff_mode_left_side
test.test_diff.DiffTest ‑ test_dataframe_diff_with_diff_mode_right_side
test.test_diff.DiffTest ‑ test_dataframe_diff_with_diff_mode_side_by_side
test.test_diff.DiffTest ‑ test_dataframe_diff_with_options
test.test_diff.DiffTest ‑ test_dataframe_diff_with_sparse_mode
test.test_diff.DiffTest ‑ test_dataframe_diffwith
test.test_diff.DiffTest ‑ test_diff_fluent_setters
test.test_diff.DiffTest ‑ test_diff_mode_consts
test.test_diff.DiffTest ‑ test_diff_options_default
test.test_diff.DiffTest ‑ test_diff_options_with_duplicate_comparators
test.test_diff.DiffTest ‑ test_diff_with_epsilon_comparator
test.test_diff.DiffTest ‑ test_differ_diff
test.test_diff.DiffTest ‑ test_differ_diff_in_diff_mode_column_by_column
test.test_diff.DiffTest ‑ test_differ_diff_in_diff_mode_left_side
test.test_diff.DiffTest ‑ test_differ_diff_in_diff_mode_right_side
test.test_diff.DiffTest ‑ test_differ_diff_in_diff_mode_side_by_side
test.test_diff.DiffTest ‑ test_differ_diff_with_changes
test.test_diff.DiffTest ‑ test_differ_diff_with_default_options
test.test_diff.DiffTest ‑ test_differ_diff_with_options
test.test_diff.DiffTest ‑ test_differ_diff_with_sparse_mode
test.test_diff.DiffTest ‑ test_differ_diffwith
test.test_histogram.HistogramTest ‑ test_histogram_with_floats
test.test_histogram.HistogramTest ‑ test_histogram_with_ints
test.test_job_description.JobDescriptionTest ‑ test_append_job_description
test.test_job_description.JobDescriptionTest ‑ test_with_job_description
test.test_jvm.PackageTest ‑ test_create_temp_dir
test.test_jvm.PackageTest ‑ test_dotnet_ticks
test.test_jvm.PackageTest ‑ test_get_jvm_classic
test.test_jvm.PackageTest ‑ test_get_jvm_connect
test.test_jvm.PackageTest ‑ test_histogram
test.test_jvm.PackageTest ‑ test_install_pip_package
test.test_jvm.PackageTest ‑ test_install_poetry_project
test.test_jvm.PackageTest ‑ test_job_description
test.test_jvm.PackageTest ‑ test_parquet
test.test_jvm.PackageTest ‑ test_with_row_numbers
test.test_package.PackageTest ‑ test_count_null
test.test_package.PackageTest ‑ test_create_temp_dir
test.test_package.PackageTest ‑ test_dotnet_ticks_to_timestamp
test.test_package.PackageTest ‑ test_dotnet_ticks_to_unix_epoch
test.test_package.PackageTest ‑ test_dotnet_ticks_to_unix_epoch_nanos
test.test_package.PackageTest ‑ test_install_pip_package
test.test_package.PackageTest ‑ test_install_pip_package_not_supported
test.test_package.PackageTest ‑ test_install_pip_package_package_not_found
test.test_package.PackageTest ‑ test_install_pip_package_unknown_argument
test.test_package.PackageTest ‑ test_install_poetry_project
test.test_package.PackageTest ‑ test_install_poetry_project_not_supported
test.test_package.PackageTest ‑ test_install_poetry_project_wrong_arguments
test.test_package.PackageTest ‑ test_timestamp_to_dotnet_ticks
test.test_package.PackageTest ‑ test_unix_epoch_dotnet_ticks
test.test_package.PackageTest ‑ test_unix_epoch_nanos_to_dotnet_ticks
test.test_parquet.ParquetTest ‑ test_parquet_block_columns
test.test_parquet.ParquetTest ‑ test_parquet_blocks
test.test_parquet.ParquetTest ‑ test_parquet_metadata
test.test_parquet.ParquetTest ‑ test_parquet_partitions
test.test_parquet.ParquetTest ‑ test_parquet_schema
test.test_row_number.RowNumberTest ‑ test_row_numbers
test.test_row_number.RowNumberTest ‑ test_row_numbers_order_desc_one_column
test.test_row_number.RowNumberTest ‑ test_row_numbers_order_desc_two_columns
test.test_row_number.RowNumberTest ‑ test_row_numbers_order_not_asc_one_column
test.test_row_number.RowNumberTest ‑ test_row_numbers_order_not_asc_two_columns
test.test_row_number.RowNumberTest ‑ test_row_numbers_order_one_column
test.test_row_number.RowNumberTest ‑ test_row_numbers_order_two_columns
test.test_row_number.RowNumberTest ‑ test_row_numbers_row_number_col_name
test.test_row_number.RowNumberTest ‑ test_row_numbers_unpersist
uk.co.gresearch.spark.GroupBySuite ‑ df.groupByKeySorted should flatMapSortedGroups
uk.co.gresearch.spark.GroupBySuite ‑ df.groupByKeySorted should flatMapSortedGroups reverse
uk.co.gresearch.spark.GroupBySuite ‑ df.groupByKeySorted should flatMapSortedGroups with partition num
uk.co.gresearch.spark.GroupBySuite ‑ df.groupByKeySorted should flatMapSortedGroups with partition num and reverse
uk.co.gresearch.spark.GroupBySuite ‑ df.groupByKeySorted should flatMapSortedGroups with state
uk.co.gresearch.spark.GroupBySuite ‑ df.groupByKeySorted should flatMapSortedGroups with tuple key
uk.co.gresearch.spark.GroupBySuite ‑ df.groupByKeySorted should flatMapSortedGroups with tuple key and state
uk.co.gresearch.spark.GroupBySuite ‑ df.groupBySorted should flatMapSortedGroups
uk.co.gresearch.spark.GroupBySuite ‑ df.groupBySorted should flatMapSortedGroups reverse
uk.co.gresearch.spark.GroupBySuite ‑ df.groupBySorted should flatMapSortedGroups with partition num
uk.co.gresearch.spark.GroupBySuite ‑ df.groupBySorted should flatMapSortedGroups with partition num and reverse
uk.co.gresearch.spark.GroupBySuite ‑ df.groupBySorted should flatMapSortedGroups with state
uk.co.gresearch.spark.GroupBySuite ‑ df.groupBySorted should flatMapSortedGroups with tuple key
uk.co.gresearch.spark.GroupBySuite ‑ df.groupBySorted should flatMapSortedGroups with tuple key and state
uk.co.gresearch.spark.GroupBySuite ‑ ds.groupByKeySorted should flatMapSortedGroups
uk.co.gresearch.spark.GroupBySuite ‑ ds.groupByKeySorted should flatMapSortedGroups reverse
uk.co.gresearch.spark.GroupBySuite ‑ ds.groupByKeySorted should flatMapSortedGroups with partition num
uk.co.gresearch.spark.GroupBySuite ‑ ds.groupByKeySorted should flatMapSortedGroups with partition num and reverse
uk.co.gresearch.spark.GroupBySuite ‑ ds.groupByKeySorted should flatMapSortedGroups with state
uk.co.gresearch.spark.GroupBySuite ‑ ds.groupByKeySorted should flatMapSortedGroups with tuple key
uk.co.gresearch.spark.GroupBySuite ‑ ds.groupByKeySorted should flatMapSortedGroups with tuple key and state
uk.co.gresearch.spark.GroupBySuite ‑ ds.groupBySorted should flatMapSortedGroups
uk.co.gresearch.spark.GroupBySuite ‑ ds.groupBySorted should flatMapSortedGroups reverse
uk.co.gresearch.spark.GroupBySuite ‑ ds.groupBySorted should flatMapSortedGroups with partition num
uk.co.gresearch.spark.GroupBySuite ‑ ds.groupBySorted should flatMapSortedGroups with partition num and reverse
uk.co.gresearch.spark.GroupBySuite ‑ ds.groupBySorted should flatMapSortedGroups with state
uk.co.gresearch.spark.GroupBySuite ‑ ds.groupBySorted should flatMapSortedGroups with tuple key
uk.co.gresearch.spark.GroupBySuite ‑ ds.groupBySorted should flatMapSortedGroups with tuple key and state
uk.co.gresearch.spark.GroupBySuite ‑ should df.groupByKey
uk.co.gresearch.spark.GroupBySuite ‑ should ds.groupByKey
uk.co.gresearch.spark.HistogramSuite ‑ histogram with double values
uk.co.gresearch.spark.HistogramSuite ‑ histogram with double values and int thresholds
uk.co.gresearch.spark.HistogramSuite ‑ histogram with double values on DataFrame
uk.co.gresearch.spark.HistogramSuite ‑ histogram with duplicate thresholds
uk.co.gresearch.spark.HistogramSuite ‑ histogram with int values
uk.co.gresearch.spark.HistogramSuite ‑ histogram with int values and double thresholds
uk.co.gresearch.spark.HistogramSuite ‑ histogram with int values on DataFrame
uk.co.gresearch.spark.HistogramSuite ‑ histogram with no aggregate columns
uk.co.gresearch.spark.HistogramSuite ‑ histogram with no thresholds
uk.co.gresearch.spark.HistogramSuite ‑ histogram with non-existing aggregate column
uk.co.gresearch.spark.HistogramSuite ‑ histogram with non-existing value columns
uk.co.gresearch.spark.HistogramSuite ‑ histogram with one aggregate column
uk.co.gresearch.spark.HistogramSuite ‑ histogram with one threshold
uk.co.gresearch.spark.HistogramSuite ‑ histogram with two aggregate columns
uk.co.gresearch.spark.SparkSuite ‑ .Net ticks to Spark timestamp / unix epoch seconds / nanoseconds
uk.co.gresearch.spark.SparkSuite ‑ Get Spark version
uk.co.gresearch.spark.SparkSuite ‑ NoopUnpersistHandle does not unpersist set DataFrame with StorageLevel(1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ NoopUnpersistHandle does not unpersist set DataFrame with StorageLevel(disk, 1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ NoopUnpersistHandle does not unpersist set DataFrame with StorageLevel(disk, memory, deserialized, 1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ NoopUnpersistHandle does not unpersist set DataFrame with StorageLevel(memory, deserialized, 1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ SilentUnpersistHandle does not throw on unpersist if no DataFrame is set
uk.co.gresearch.spark.SparkSuite ‑ SilentUnpersistHandle does not throw on unpersist with blocking if no DataFrame is set
uk.co.gresearch.spark.SparkSuite ‑ SilentUnpersistHandle does unpersist set DataFrame with StorageLevel(1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ SilentUnpersistHandle does unpersist set DataFrame with StorageLevel(disk, 1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ SilentUnpersistHandle does unpersist set DataFrame with StorageLevel(disk, memory, deserialized, 1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ SilentUnpersistHandle does unpersist set DataFrame with StorageLevel(memory, deserialized, 1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ SilentUnpersistHandle throws on setting DataFrame twice
uk.co.gresearch.spark.SparkSuite ‑ Spark temp dir
uk.co.gresearch.spark.SparkSuite ‑ Spark timestamp to .Net ticks
uk.co.gresearch.spark.SparkSuite ‑ Unix epoch nanos to .Net ticks
uk.co.gresearch.spark.SparkSuite ‑ Unix epoch to .Net ticks
uk.co.gresearch.spark.SparkSuite ‑ UnpersistHandle does unpersist set DataFrame with StorageLevel(1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ UnpersistHandle does unpersist set DataFrame with StorageLevel(disk, 1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ UnpersistHandle does unpersist set DataFrame with StorageLevel(disk, memory, deserialized, 1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ UnpersistHandle does unpersist set DataFrame with StorageLevel(memory, deserialized, 1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ UnpersistHandle throws on setting DataFrame twice
uk.co.gresearch.spark.SparkSuite ‑ UnpersistHandle throws on unpersist if no DataFrame is set
uk.co.gresearch.spark.SparkSuite ‑ UnpersistHandle throws on unpersist with blocking if no DataFrame is set
uk.co.gresearch.spark.SparkSuite ‑ append job description
uk.co.gresearch.spark.SparkSuite ‑ backticks
uk.co.gresearch.spark.SparkSuite ‑ call dataframe-to-dataframe transformation
uk.co.gresearch.spark.SparkSuite ‑ call dataframe-to-dataset transformation
uk.co.gresearch.spark.SparkSuite ‑ call dataset-to-dataframe transformation
uk.co.gresearch.spark.SparkSuite ‑ call dataset-to-dataset transformation
uk.co.gresearch.spark.SparkSuite ‑ call on false condition dataframe-to-dataframe transformation
uk.co.gresearch.spark.SparkSuite ‑ call on false condition dataset-to-dataset transformation
uk.co.gresearch.spark.SparkSuite ‑ call on false condition either dataframe-to-dataframe transformation
uk.co.gresearch.spark.SparkSuite ‑ call on false condition either dataframe-to-dataset transformation
uk.co.gresearch.spark.SparkSuite ‑ call on false condition either dataset-to-dataframe transformation
uk.co.gresearch.spark.SparkSuite ‑ call on false condition either dataset-to-dataset transformation
uk.co.gresearch.spark.SparkSuite ‑ call on true condition dataframe-to-dataframe transformation
uk.co.gresearch.spark.SparkSuite ‑ call on true condition dataset-to-dataset transformation
uk.co.gresearch.spark.SparkSuite ‑ call on true condition either dataframe-to-dataframe transformation
uk.co.gresearch.spark.SparkSuite ‑ call on true condition either dataframe-to-dataset transformation
uk.co.gresearch.spark.SparkSuite ‑ call on true condition either dataset-to-dataframe transformation
uk.co.gresearch.spark.SparkSuite ‑ call on true condition either dataset-to-dataset transformation
uk.co.gresearch.spark.SparkSuite ‑ count_null
uk.co.gresearch.spark.SparkSuite ‑ global row number allows desc order
uk.co.gresearch.spark.SparkSuite ‑ global row number allows to unpersist with StorageLevel(1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ global row number allows to unpersist with StorageLevel(disk, 1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ global row number allows to unpersist with StorageLevel(disk, memory, deserialized, 1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ global row number allows to unpersist with StorageLevel(disk, memory, offheap, 1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ global row number allows to unpersist with StorageLevel(memory, deserialized, 1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ global row number preserves order
uk.co.gresearch.spark.SparkSuite ‑ global row number respects order
uk.co.gresearch.spark.SparkSuite ‑ global row number supports multiple order columns
uk.co.gresearch.spark.SparkSuite ‑ global row number with StorageLevel(1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ global row number with StorageLevel(disk, 1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ global row number with StorageLevel(disk, memory, deserialized, 1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ global row number with StorageLevel(disk, memory, offheap, 1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ global row number with StorageLevel(memory, deserialized, 1 replicas)
uk.co.gresearch.spark.SparkSuite ‑ global row number with custom row_number column
uk.co.gresearch.spark.SparkSuite ‑ global row number with existing row_number column
uk.co.gresearch.spark.SparkSuite ‑ global row number with internal column names
uk.co.gresearch.spark.SparkSuite ‑ on false condition call either writer-to-unit methods
uk.co.gresearch.spark.SparkSuite ‑ on false condition call either writer-to-writer methods
uk.co.gresearch.spark.SparkSuite ‑ on true condition call either writer-to-unit methods
uk.co.gresearch.spark.SparkSuite ‑ on true condition call either writer-to-writer methods
uk.co.gresearch.spark.SparkSuite ‑ when false call dataframe-to-dataframe transformation
uk.co.gresearch.spark.SparkSuite ‑ when false call dataset-to-dataset transformation
uk.co.gresearch.spark.SparkSuite ‑ when true call dataframe-to-dataframe transformation
uk.co.gresearch.spark.SparkSuite ‑ when true call dataset-to-dataset transformation
uk.co.gresearch.spark.SparkSuite ‑ with existing job description
uk.co.gresearch.spark.SparkSuite ‑ with existing job description if not set
uk.co.gresearch.spark.SparkSuite ‑ with job description
uk.co.gresearch.spark.SparkSuite ‑ without job description
uk.co.gresearch.spark.WritePartitionedSuite ‑ write dataframe
uk.co.gresearch.spark.WritePartitionedSuite ‑ write partitionedBy requires caching with AQE enabled
uk.co.gresearch.spark.WritePartitionedSuite ‑ write partitionedBy requires no caching with AQE disabled
uk.co.gresearch.spark.WritePartitionedSuite ‑ write with desc partition order
uk.co.gresearch.spark.WritePartitionedSuite ‑ write with more partition columns
uk.co.gresearch.spark.WritePartitionedSuite ‑ write with one partition
uk.co.gresearch.spark.WritePartitionedSuite ‑ write with one partition column
uk.co.gresearch.spark.WritePartitionedSuite ‑ write with partition order
uk.co.gresearch.spark.WritePartitionedSuite ‑ write with two partition column
uk.co.gresearch.spark.WritePartitionedSuite ‑ write with write projection
uk.co.gresearch.spark.diff.AppSuite ‑ run app with file and hive table
uk.co.gresearch.spark.diff.AppSuite ‑ run app with filter [C]
uk.co.gresearch.spark.diff.AppSuite ‑ run app with filter [D]
uk.co.gresearch.spark.diff.AppSuite ‑ run app with filter [I,C,D]
uk.co.gresearch.spark.diff.AppSuite ‑ run app with filter [I]
uk.co.gresearch.spark.diff.AppSuite ‑ run app with filter [N]
uk.co.gresearch.spark.diff.AppSuite ‑ run app with unknown filter
uk.co.gresearch.spark.diff.AppSuite ‑ run app writing stats
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ DiffComparator.default
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ DiffComparator.duration
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ DiffComparator.epsilon
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ DiffComparator.equiv with any
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ DiffComparator.equiv with encoder
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ DiffComparator.equiv with type
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ DiffComparator.nullSafeEqual
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ absolute epsilon comparator (exclusive)
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ absolute epsilon comparator (inclusive)
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ changeset accounts for comparators
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ diff with custom comparator - codegen enabled=false
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ diff with custom comparator - codegen enabled=true
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ duration comparator not supported
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ duration comparator with date (exclusive)
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ duration comparator with date (inclusive)
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ duration comparator with time (exclusive)
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ duration comparator with time (inclusive)
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ map comparator typed - keyOrderSensitive=false - codegen enabled=false
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ map comparator typed - keyOrderSensitive=false - codegen enabled=true
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ map comparator typed - keyOrderSensitive=true - codegen enabled=false
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ map comparator typed - keyOrderSensitive=true - codegen enabled=true
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ map comparator untyped - keyOrderSensitive=false - codegen enabled=false
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ map comparator untyped - keyOrderSensitive=false - codegen enabled=true
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ map comparator untyped - keyOrderSensitive=true - codegen enabled=false
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ map comparator untyped - keyOrderSensitive=true - codegen enabled=true
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ null-aware comparator
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ relative epsilon comparator (exclusive)
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ relative epsilon comparator (inclusive)
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ whitespace agnostic string comparator
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ with comparator - any equiv for column name
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ with comparator - default any equiv
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ with comparator - default diff comparator
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ with comparator - default encoder equiv
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ with comparator - default typed equiv
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ with comparator - diff comparator for name
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ with comparator - diff comparator for type
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ with comparator - encoder equiv
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ with comparator - encoder equiv for column name
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ with comparator - equiv encoder for column name
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ with comparator - typed diff comparator
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ with comparator - typed diff comparator for type
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ with comparator - typed equiv for type
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ with comparator of incompatible type - diff comparator
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ with comparator of incompatible type - encoder equiv
uk.co.gresearch.spark.diff.DiffComparatorSuite ‑ with comparator of incompatible type - typed equiv
uk.co.gresearch.spark.diff.DiffJavaTests ‑ testDiff
uk.co.gresearch.spark.diff.DiffJavaTests ‑ testDiffAs
uk.co.gresearch.spark.diff.DiffJavaTests ‑ testDiffIgnoredColumn
uk.co.gresearch.spark.diff.DiffJavaTests ‑ testDiffMultipleKeys
uk.co.gresearch.spark.diff.DiffJavaTests ‑ testDiffNoKey
uk.co.gresearch.spark.diff.DiffJavaTests ‑ testDiffOfWith
uk.co.gresearch.spark.diff.DiffJavaTests ‑ testDiffSingleKey
uk.co.gresearch.spark.diff.DiffJavaTests ‑ testDiffWithComparators
uk.co.gresearch.spark.diff.DiffJavaTests ‑ testDiffWithOptions
uk.co.gresearch.spark.diff.DiffJavaTests ‑ testDiffer
uk.co.gresearch.spark.diff.DiffJavaTests ‑ testDifferWithIgnored
uk.co.gresearch.spark.diff.DiffOptionsSuite ‑ diff options diff value
uk.co.gresearch.spark.diff.DiffOptionsSuite ‑ diff options left and right prefixes
uk.co.gresearch.spark.diff.DiffOptionsSuite ‑ diff options with change column name same as diff column
uk.co.gresearch.spark.diff.DiffOptionsSuite ‑ diff options with comparators
uk.co.gresearch.spark.diff.DiffOptionsSuite ‑ diff options with duplicate comparator - multiple columns
uk.co.gresearch.spark.diff.DiffOptionsSuite ‑ diff options with duplicate comparator - multiple types
uk.co.gresearch.spark.diff.DiffOptionsSuite ‑ diff options with duplicate comparator - single column
uk.co.gresearch.spark.diff.DiffOptionsSuite ‑ diff options with duplicate comparator - single type
uk.co.gresearch.spark.diff.DiffOptionsSuite ‑ diff options with empty diff column name
uk.co.gresearch.spark.diff.DiffOptionsSuite ‑ diff options with typed diff comparator for other data type
uk.co.gresearch.spark.diff.DiffOptionsSuite ‑ fluent methods of diff options
uk.co.gresearch.spark.diff.DiffSuite ‑ diff DataFrames
uk.co.gresearch.spark.diff.DiffSuite ‑ diff as U
uk.co.gresearch.spark.diff.DiffSuite ‑ diff as U with encoder
uk.co.gresearch.spark.diff.DiffSuite ‑ diff as U with encoder and custom options
uk.co.gresearch.spark.diff.DiffSuite ‑ diff as U with extra column
uk.co.gresearch.spark.diff.DiffSuite ‑ diff as U with left-side diff mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff as U with right-side diff mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff as U with subset of columns
uk.co.gresearch.spark.diff.DiffSuite ‑ diff column order
uk.co.gresearch.spark.diff.DiffSuite ‑ diff dataframe with duplicate columns
uk.co.gresearch.spark.diff.DiffSuite ‑ diff of empty schema
uk.co.gresearch.spark.diff.DiffSuite ‑ diff of non-existing id column
uk.co.gresearch.spark.diff.DiffSuite ‑ diff similar with ignored column and different number of columns
uk.co.gresearch.spark.diff.DiffSuite ‑ diff similar with ignored columns
uk.co.gresearch.spark.diff.DiffSuite ‑ diff similar with ignored columns and empty schema
uk.co.gresearch.spark.diff.DiffSuite ‑ diff similar with ignored columns case-insensitive
uk.co.gresearch.spark.diff.DiffSuite ‑ diff similar with ignored columns case-sensitive
uk.co.gresearch.spark.diff.DiffSuite ‑ diff similar with ignored columns of different type
uk.co.gresearch.spark.diff.DiffSuite ‑ diff where case-insensitive non-id column produces change column name
uk.co.gresearch.spark.diff.DiffSuite ‑ diff where case-insensitive non-id column produces diff column name
uk.co.gresearch.spark.diff.DiffSuite ‑ diff where case-insensitive non-id column produces id column name
uk.co.gresearch.spark.diff.DiffSuite ‑ diff where case-sensitive non-id column produces non-conflicting change column name
uk.co.gresearch.spark.diff.DiffSuite ‑ diff where case-sensitive non-id column produces non-conflicting diff column name
uk.co.gresearch.spark.diff.DiffSuite ‑ diff where case-sensitive non-id column produces non-conflicting id column name
uk.co.gresearch.spark.diff.DiffSuite ‑ diff where non-id column produces change column name
uk.co.gresearch.spark.diff.DiffSuite ‑ diff where non-id column produces diff column name
uk.co.gresearch.spark.diff.DiffSuite ‑ diff where non-id column produces id column name
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with all id columns
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with case-insensitive column names
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with case-sensitive column names
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with change column
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with change column name in id columns
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with change column name in non-id columns
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with change column without id columns
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with column-by-column and sparse mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with column-by-column diff mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with custom diff options
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with different column names
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with different nullability
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with different number of columns
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with different types
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with dot in id column
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with dot in value column
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with dots in change column
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with dots in diff column
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with dots in prefixes
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with id column diff in T
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with ignored and change columns
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with ignored columns
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with ignored columns and column-by-column diff mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with ignored columns and left-side diff mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with ignored columns and right-side diff mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with ignored columns and side-by-side diff mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with ignored columns case-insensitive
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with ignored columns case-sensitive
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with ignored columns of different types
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with ignored columns, column-by-column diff and sparse mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with ignored columns, left-side diff and sparse mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with ignored columns, right-side diff and sparse mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with ignored columns, side-by-side diff and sparse mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with left side and sparse mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with left-side diff mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with left-side diff mode and change column name in value columns
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with left-side diff mode and diff column name in value columns
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with left-side diff mode and dot in value column
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with left-side mode where case-insensitive non-id column would produce diff column name
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with left-side mode where non-id column would produce diff column name
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with no id column
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with no id columns ids taken from left
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with non-id column diff in T
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with null id values
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with null values
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with one ID column case-insensitive
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with one id column
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with one id column case-sensitive
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with output columns in T
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with right side and sparse mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with right-side diff mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with right-side diff mode and change column name in value columns
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with right-side diff mode and diff column name in value columns
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with right-side diff mode and dot in value column
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with right-side mode where case-insensitive non-id column would produce diff column name
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with right-side mode where non-id column would produce diff column name
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with side-by-side and sparse mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with side-by-side diff mode
uk.co.gresearch.spark.diff.DiffSuite ‑ diff with two id columns
uk.co.gresearch.spark.diff.DiffSuite ‑ diffWith
uk.co.gresearch.spark.diff.DiffSuite ‑ diffWith left-prefixed id
uk.co.gresearch.spark.diff.DiffSuite ‑ diffWith right-prefixed id
uk.co.gresearch.spark.diff.DiffSuite ‑ distinct prefix for
uk.co.gresearch.spark.diff.examples.Examples ‑ examples
uk.co.gresearch.spark.diff.examples.Examples ‑ issue
uk.co.gresearch.spark.group.GroupSuite ‑ GroupIterator should iterate only over current key for multiple keys and multiple values
uk.co.gresearch.spark.group.GroupSuite ‑ GroupIterator should iterate only over current key for multiple keys and single value
uk.co.gresearch.spark.group.GroupSuite ‑ GroupIterator should iterate only over current key for null key
uk.co.gresearch.spark.group.GroupSuite ‑ GroupIterator should iterate only over current key for single key and multiple values
uk.co.gresearch.spark.group.GroupSuite ‑ GroupIterator should iterate only over current key for single key and single value
uk.co.gresearch.spark.group.GroupSuite ‑ GroupIterator should not work with empty iterator
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None key and fully consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None key and multiple calls to hasNext
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None key and partially consumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None key and partially consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None key and partially consumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None key and partially consumed second group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None key and unconsumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None key and unconsumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None key and unconsumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None key and unconsumed second group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None values and fully consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None values and multiple calls to hasNext
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None values and partially consumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None values and partially consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None values and partially consumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None values and partially consumed second group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None values and unconsumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None values and unconsumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None values and unconsumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with None values and unconsumed second group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with empty iterator and fully consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with empty iterator and multiple calls to hasNext
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with empty iterator and partially consumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with empty iterator and partially consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with empty iterator and partially consumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with empty iterator and partially consumed second group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with empty iterator and unconsumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with empty iterator and unconsumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with empty iterator and unconsumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with empty iterator and unconsumed second group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many 1-element groups and fully consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many 1-element groups and multiple calls to hasNext
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many 1-element groups and partially consumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many 1-element groups and partially consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many 1-element groups and partially consumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many 1-element groups and partially consumed second group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many 1-element groups and unconsumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many 1-element groups and unconsumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many 1-element groups and unconsumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many 1-element groups and unconsumed second group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many groups and fully consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many groups and multiple calls to hasNext
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many groups and partially consumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many groups and partially consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many groups and partially consumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many groups and partially consumed second group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many groups and unconsumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many groups and unconsumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many groups and unconsumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with many groups and unconsumed second group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null key and fully consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null key and multiple calls to hasNext
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null key and partially consumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null key and partially consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null key and partially consumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null key and partially consumed second group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null key and unconsumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null key and unconsumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null key and unconsumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null key and unconsumed second group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null values and fully consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null values and multiple calls to hasNext
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null values and partially consumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null values and partially consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null values and partially consumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null values and partially consumed second group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null values and unconsumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null values and unconsumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null values and unconsumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with null values and unconsumed second group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one 1-element groups and fully consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one 1-element groups and multiple calls to hasNext
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one 1-element groups and partially consumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one 1-element groups and partially consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one 1-element groups and partially consumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one 1-element groups and partially consumed second group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one 1-element groups and unconsumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one 1-element groups and unconsumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one 1-element groups and unconsumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one 1-element groups and unconsumed second group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one group and fully consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one group and multiple calls to hasNext
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one group and partially consumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one group and partially consumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one group and partially consumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one group and partially consumed second group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one group and unconsumed first group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one group and unconsumed groups
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one group and unconsumed last group
uk.co.gresearch.spark.group.GroupSuite ‑ GroupedIterator should work with one group and unconsumed second group
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet block columns (parallelism=1)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet block columns (parallelism=2)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet block columns (parallelism=8)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet block columns (parallelism=None)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet blocks (parallelism=1)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet blocks (parallelism=2)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet blocks (parallelism=8)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet blocks (parallelism=None)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet metadata (parallelism=1)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet metadata (parallelism=2)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet metadata (parallelism=8)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet metadata (parallelism=None)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (1 bytes)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (1024 bytes) (parallelism=1)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (1024 bytes) (parallelism=2)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (1024 bytes) (parallelism=8)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (1024 bytes) (parallelism=None)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (127 bytes)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (13 bytes)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (19 bytes)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (251 bytes)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (29 bytes)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (3 bytes)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (512 bytes) (parallelism=1)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (512 bytes) (parallelism=2)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (512 bytes) (parallelism=8)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (512 bytes) (parallelism=None)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (61 bytes)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (7 bytes)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (8192 bytes) (parallelism=1)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (8192 bytes) (parallelism=2)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (8192 bytes) (parallelism=8)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (8192 bytes) (parallelism=None)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (default bytes) (parallelism=1)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (default bytes) (parallelism=2)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (default bytes) (parallelism=8)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet partitions (default bytes) (parallelism=None)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet schema (parallelism=1)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet schema (parallelism=2)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet schema (parallelism=8)
uk.co.gresearch.spark.parquet.ParquetSuite ‑ read parquet schema (parallelism=None)
uk.co.gresearch.test.SparkJavaTests ‑ testBackticks
uk.co.gresearch.test.SparkJavaTests ‑ testHistogram
uk.co.gresearch.test.SparkJavaTests ‑ testHistogramWithAggColumn
uk.co.gresearch.test.SparkJavaTests ‑ testRowNumbers
uk.co.gresearch.test.SparkJavaTests ‑ testRowNumbersColumnName
uk.co.gresearch.test.SparkJavaTests ‑ testRowNumbersOrderDesc
uk.co.gresearch.test.SparkJavaTests ‑ testRowNumbersOrderOneColumn
uk.co.gresearch.test.SparkJavaTests ‑ testRowNumbersOrderTwoColumns
uk.co.gresearch.test.SparkJavaTests ‑ testRowNumbersStorageLevelAndUnpersist
uk.co.gresearch.test.SparkJavaTests ‑ testRowNumbersUnpersist
View more details on GitHub Actions