feat: raise error in from_parquet when df is empty

[ireneisdoomed]

This PR raises an error in Dataset.from_parquet if the file is found to be empty, preventing silent failures in data pipelines.

isEmpty() works by invoking limit(1).count(), which means that forces the evaluation of this operation in the Spark plan. I don't think this will have a major impact in the performance, and reading operations are done very early in the pipeline. And, if anything, the step will fail faster.

I've come across this bug a few times, most recently when accessing the credible set folder for L2G. I think that due to incompatible schemas from various sources, querying this folder returned an empty dataframe.

[tskir]

Thank you @ireneisdoomed! Looks like a very good check to have

[ireneisdoomed]

For the record, after this change the GWASCatalog step went from ~27' to 42'.

[d0choa]

@ireneisdoomed we are producing empty dataframes as part of the GWAS catalog harmonisation. Rows that don't comply with the standards (e.g. no beta, no position, etc.) are filtered out resulting on empty dataframes. SummaryStats -> StudyLocus can result on non-significant associations that could create an empty data frame.

[ireneisdoomed]

@d0choa I don't necessarily see the problem?

1. If the path points to all the harmonised results / all studylocus, the resulting df won't be empty.
2. If it points to a single harmonised study with no significant associations, I think it is positive to flag they are empty.