Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Spark] Make delta.dataSkippingStatsColumns more lenient for nested columns #2850

Open
wants to merge 18 commits into
base: master
Choose a base branch
from
Open

[Spark] Make delta.dataSkippingStatsColumns more lenient for nested columns #2850

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
a9412bc
Starting
Kimahriman Mar 29, 2024
a864205
Remove another test
Kimahriman Mar 29, 2024
975f20b
Comment out irrelevant tests
Kimahriman Apr 1, 2024
6411e7a
Try to fix scalastyle
Kimahriman Apr 1, 2024
e68028d
Add duplicate col checking back
Kimahriman Apr 3, 2024
cbb0447
Remove commented out
Kimahriman Apr 4, 2024
3759afd
Add test for nested complex types
Kimahriman Apr 4, 2024
329a571
Clean up commented out code
Kimahriman Apr 4, 2024
b911d6a
Merge branch 'master' into lenient-stats-cols
Kimahriman Aug 16, 2024
0ea2fc8
Update spark/src/main/scala/org/apache/spark/sql/delta/stats/Statisti…
Kimahriman Aug 17, 2024
81c60aa
Update variable name and add double nested struct to test
Kimahriman Aug 17, 2024
6068eb0
Fix style
Kimahriman Aug 17, 2024
60978a8
Merge branch 'master' into lenient-stats-cols
Kimahriman Aug 18, 2024
bb86d96
Add more hits tests
Kimahriman Aug 18, 2024
0c24d44
Add more valid type tests
Kimahriman Aug 18, 2024
9bd8159
Change description
Kimahriman Aug 18, 2024
86c148f
Add warning log
Kimahriman Aug 19, 2024
25e6faa
Merge branch 'master' into lenient-stats-cols
Kimahriman Sep 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions spark/src/main/scala/org/apache/spark/sql/delta/stats/StatisticsCollection.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -458,15 +458,18 @@ object StatisticsCollection extends DeltaCommand {
* @param name The name of the data skipping column for validating data type.
* @param dataType The data type of the data skipping column.
* @param columnPaths The column paths of all valid fields.
* @param inStruct Whether we are inside a struct already, in which case we don't care about type
Kimahriman marked this conversation as resolved.
Show resolved Hide resolved
*/
private def validateDataSkippingType(
name: String,
dataType: DataType,
columnPaths: ArrayBuffer[String]): Unit = dataType match {
columnPaths: ArrayBuffer[String],
inStruct: Boolean = false): Unit = dataType match {
Kimahriman marked this conversation as resolved.
Show resolved Hide resolved
case s: StructType =>
s.foreach { field =>
validateDataSkippingType(name + "." + field.name, field.dataType, columnPaths)
validateDataSkippingType(name + "." + field.name, field.dataType, columnPaths, true)
Kimahriman marked this conversation as resolved.
Show resolved Hide resolved
}
case _ if inStruct => columnPaths.append(name)
case SkippingEligibleDataType(_) => columnPaths.append(name)
case _ =>
throw new DeltaIllegalArgumentException(
Expand Down
21 changes: 21 additions & 0 deletions spark/src/test/scala/org/apache/spark/sql/delta/stats/DataSkippingDeltaTests.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -599,6 +599,27 @@ trait DataSkippingDeltaTestsBase extends DeltaExcludedBySparkVersionTestMixinShi
deltaStatsColNamesOpt = Some("b.c")
)

testSkipping(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider double struct test?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a double nested struct as well

"indexed column names - naming a nested column allows nested complex types",
"""{
"a": {
"b": [1, 2, 3],
"c": [4, 5, 6],
"d": 7,
"e": 8
},
"f": 9
}""".replace("\n", ""),
hits = Seq(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we have hit tests for elements inside struct and the double struct as well?

Btw click on re-request review so that I get notified.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added more hit tests

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that when we are inside struct, we gather stats for the valid datatypes inside the struct, but the invalid ones we still don't right? Would we want to test that we still don't skip the invalid datatypes inside the struct?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, if we can check another skipping valid type beside integer, like timestamp, string, ... that would be great

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically null counts are collected, but they can't be used for skipping. We could add that to the test, but seems kind of out of scope because you can't do min/max things with arrays/maps anyway, so it'd have to be like an element contains or something

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there are other non-skipping eligible datatypes than lists datatypes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how to specify a Binary type in the inferred JSON. Is that the only non-complex type?

"f < 0"
),
misses = Seq(
"a.d < 0",
"a.e < 0"
),
deltaStatsColNamesOpt = Some("a")
)

testSkipping(
"indexed column names - index only a subset of leaf columns",
"""{
Expand Down
20 changes: 1 addition & 19 deletions spark/src/test/scala/org/apache/spark/sql/delta/stats/StatsCollectionSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -499,8 +499,7 @@ class StatsCollectionSuite
("BINARY", "BinaryType"),
("BOOLEAN", "BooleanType"),
("ARRAY<TINYINT>", "ArrayType(ByteType,true)"),
Kimahriman marked this conversation as resolved.
Show resolved Hide resolved
("MAP<DATE, INT>", "MapType(DateType,IntegerType,true)"),
("STRUCT<c60:INT, c61:ARRAY<INT>>", "ArrayType(IntegerType,true)")
("MAP<DATE, INT>", "MapType(DateType,IntegerType,true)")
Kimahriman marked this conversation as resolved.
Show resolved Hide resolved
).foreach { case (invalidType, typename) =>
val tableName1 = "delta_table_1"
val tableName2 = "delta_table_2"
Expand Down Expand Up @@ -541,16 +540,6 @@ class StatsCollectionSuite
exceptOne.getErrorClass == "DELTA_COLUMN_DATA_SKIPPING_NOT_SUPPORTED_TYPE" &&
exceptOne.getMessageParametersArray.toSeq == Seq(columnName, typename)
)
val exceptTwo = intercept[DeltaIllegalArgumentException] {
sql(
s"create table $tableName1 (c1 long, c2 STRUCT<c20:INT, c21:$invalidType>) " +
s"using delta TBLPROPERTIES('delta.dataSkippingStatsColumns' = 'c2')"
)
}
assert(
exceptTwo.getErrorClass == "DELTA_COLUMN_DATA_SKIPPING_NOT_SUPPORTED_TYPE" &&
exceptTwo.getMessageParametersArray.toSeq == Seq(columnName, typename)
)
sql(s"create table $tableName2 (c1 long, c2 STRUCT<c20:INT, c21:$invalidType>) using delta")
val exceptThree = interceptWithUnwrapping[DeltaIllegalArgumentException] {
sql(
Expand All @@ -561,13 +550,6 @@ class StatsCollectionSuite
exceptThree.getErrorClass == "DELTA_COLUMN_DATA_SKIPPING_NOT_SUPPORTED_TYPE" &&
exceptThree.getMessageParametersArray.toSeq == Seq(columnName, typename)
)
val exceptFour = interceptWithUnwrapping[DeltaIllegalArgumentException] {
sql(s"ALTER TABLE $tableName2 SET TBLPROPERTIES('delta.dataSkippingStatsColumns'='c2')")
}
assert(
exceptFour.getErrorClass == "DELTA_COLUMN_DATA_SKIPPING_NOT_SUPPORTED_TYPE" &&
exceptFour.getMessageParametersArray.toSeq == Seq(columnName, typename)
)
}
}
}
Expand Down
Loading