-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix: make sure that no inferred schemas ever hit a delta lake
This commit fixes a couple oversights with inferred schemas: - The type might clash with the wide schema we have per-spec, even though the types are really compatible (inferred int for a float field does happen) - The wrong type might get through unnoticed if it is a deep field that our per-spec schema didn't catch. Here's a summary of changes to make that happen: - Drop all use of pandas. It's too loose with the types. Instead, switch to pyarrow schemas which were used under the covers of pandas anyway. - Add schema earlier in the process (at Task batching time, not at Formatter writing time). This means that all formatters get to see the same nice schema, though only deltalake uses it.
- Loading branch information
Showing
30 changed files
with
620 additions
and
447 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
"""Support for talking to FHIR servers & handling the FHIR spec""" | ||
|
||
from .fhir_client import FhirClient, create_fhir_client_for_cli | ||
from .fhir_schemas import create_spark_schema_for_resource | ||
from .fhir_schemas import pyarrow_schema_from_resource_batch | ||
from .fhir_utils import download_reference, get_docref_note, ref_resource, unref_resource |
Oops, something went wrong.