Validate and add support for Spark 4.0.0

[fdosani]

Spark 4.0.0 is out now. Should spend some time to understand any differences and test if we are compatible with the latest version: https://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-35-to-40

[fdosani]

Documenting some of the changes I'm seeing so far:

• https://spark.apache.org/docs/4.0.0/sql-ref-ansi-compliance.html#cast (spark.sql.ansi.enabled: false needs to be set it seems)