v0.5.0

The released docker image is available here.

What's Changed

Major changes

• Added support for Bulk Export API as a new resource fetching method by @chandrashekar-s; #1031, #1082, #1110, #1146
• Documentation website created at github.com/google/fhir-data-pipes by @fredhersch; #1088
• ViewDefinition-to-Parquet by @itsiggs ; #1130, #1130, #1158, #1169, #1171

List of all changes (excluding dependabot's)

• Added test_duckdb.ipynb by @bashir2 in #1064
• Added the schema doc by @bashir2 in #1087
• Added README and fixed docs style issues; removed site by @bashir2 in #1096
• Added the missing encounter_flat_view.json by @bashir2 in #1097
• Migrate the unit test annotations to Junit 5 by @chandrashekar-s in #1086
• Upgrade hapi Fhir version to 7.2.1 by @chandrashekar-s in #971
• Upgrade spring framework to 3.2.2 by @chandrashekar-s in #1108
• Added s3 in schema parsing by @bashir2 in #1111
• Update Mkdocs Documentation by @itsiggs in #1110
• Optimize build time by @chandrashekar-s in #1098
• Re-throw FHIR client exceptions by @bashir2 in #1112
• Added cacheBundleForParquetWrites option by @bashir2 in #1131
• Remove path variable when retrieving the error log file. by @chandrashekar-s in #1139
• Change canonical tables to views by @bashir2 in #1189
• Fixed an edge case for incremental run failures on cloud file-systems by @bashir2 in #1185
• Added fhirVersion to ViewDefinitions by @bashir2 in #1184
• Updates tests for converter idempotency by @bashir2 in #1205
• Delete resources in the sink DB and FHIR server by @chandrashekar-s in #1183

New Contributors

• @itsiggs made their first contribution in #1110

Full Changelog: 0.4.0...0.5.0

v0.4.0

The released docker image is available here.

What's Changed

Major changes

• A series of improvements to how profiles/extensions are handled, namely fixing #560 #558 #753 #960 #980 #1027
• An option for choosing FHIR version #400
• Adding SQL-on-FHIR-v2 tests and becoming an official implementation of the spec #954
• A simple authoring environment for SQL-on-FHIR-v2 ViewDefinition #996
• Sample flat ViewDefinitions for 10+ resource types #978
• Support for S3 file system #1011
• Exposing Sink FHIR server feature in the controller #825

List of all changes (excluding dependabot's)

• added codecov GH action by @bashir2 in #973
• updated codecov badge by @bashir2 in #979
• Support for custom FHIR profile by @chandrashekar-s in #924
• Added SQL-on-FHIR v2 repo tests by @bashir2 in #988
• Added a simple ViewDefinition edit environment by @bashir2 in #996
• Support for multiple FHIR profile extensions by @chandrashekar-s in #986
• Adds ViewDefinition for all sample flat views by @bashir2 in #1004
• Added support to read FHIR resources from NDJson file by @chandrashekar-s in #1009
• Avoid resetting connections to mitigate thrift server errors. by @bashir2 in #1017
• Merges StructureDefinition paths, adds recursiveDepth option, and makes ID columns consistent by @bashir2 in #1026
• Generated SQL-on-FHIR-v2 test results and updated tests by @bashir2 in #1032
• Added fhirVersion to SoFv2 tests and fixed report format by @bashir2 in #1033
• Fixes SoFv2 test result format by @bashir2 in #1045
• Fixes Dafaflow Parquet issue by using finishBundle by @bashir2 in #1047
• Updated tests from SoFv2 repo by @bashir2 in #1046
• Changes to fix issues of missing extensions and others by @chandrashekar-s in #1050
• Added S3 dep and some version updates by @bashir2 in #1066
• Fix the issue of DirectRunner generating too many Parquet files. by @bashir2 in #1070
• Added FHIR Sink Config Properties to the Pipeline Controller by @mozzy11 in #947
• Turn sinkFhirServerUrl off by default but on in e2e by @bashir2 in #1075

Full Changelog: 0.3.0...0.4.0

v0.3.0

The released docker image is available here.

Major features

• Support for SQL-on-FHIR-v2 views in relational DBs; issues #821 #822 #916
• Several improvements in memory management/requirements; issues #777 #823 #824 #948
• Support for OAuth based authentication in the pipelines/controller; issue #587
• Sample Spark SQL flat views added for 10+ resources; issues #820

Bug fixes/minor features

• Fix resource ID's mismatch between original FHIR resource and the corresponding resource in the parquet file in HAPI JDBC mode. by @vbothe23 in #754
• Handling large resource parsing exception using DataFormatException by @suriyan3 in #756
• Fixes missing QuestionnaireResponse answer fields by @bashir2 in #759
• Fixed bugs in recursion depth and added a path example by @bashir2 in #761
• Add user name and password to controller config by @LJNIC in #781
• Added sample flat views for 12 resources by @bashir2 in #844
• Added the option for automatic registration of views. by @bashir2 in #850
• Register micrometer gauges as strong references by @chandrashekar-s in #852
• Addressed "critical" docker image vulnerabilities by @bashir2 in #860
• Prevent creation of resource tables for erroneous DWH dirs by @bashir2 in #866
• Changes to support the Windows OS platform by @chandrashekar-s in #868
• Do not display the View Raw Logs button when the error file is not available by @chandrashekar-s in #870
• adds uuid to the list of primitive types by @bashir2 in #937
• Fixes the changes in handling of text vs zipped resources in recent HAPI versions by @muhammad-levi in #926
• Deprecation of maxWorkers param by @chandrashekar-s in #951
• replaces the spinner GIF with CSS by @bashir2 in #953
• Update Hibernate.dialect for latest HAPI version by @chandrashekar-s in #970
• Some minor pom clean-ups by @bashir2 in #962
• Minor change/fix in default memory by @bashir2 in #972

New Contributors

• @suriyan3 made their first contribution in #756
• @vbothe23 made their first contribution in #754
• @LJNIC made their first contribution in #781
• @muhammad-levi made their first contribution in #926

Full Changelog: 0.1.0...0.3.0

First beta release

This is the first beta release. The released docker image is available here.

Main features

• Pipelines to fetch resources from a FHIR server and transform them into Analytics friendly formats:
  • Input modes include: FHIR-search API, direct JDBC connection (for HAPI FHIR or OpenMRS), and JSON files.
  • Output modes include: Parquet files, another FHIR server, or PostgreSQL JSONB based tables.
  • Pipelines implemented using Apache Beam making it possible to run on different local or cluster environments such as Apache Flink, Apache Spark or Dataflow.
  • Support bulk fetch of all resources and incremental updates.
• A controller to manage and monitor continuous updates of the data warehouse:
  • Simplifies monitoring of pipelines, shows progress, vital pipeline execution metrics, error logs, etc.
  • Automatically updates table definitions if a Spark Thrift-server is configured.
• Updated version of the Bunsen library to convert FHIR resources to SQL-on-FHIR based records.
  • Added FHIR R4 support.
  • High fidelity with SQL-on-FHIR on BigQuery.
• A set of docker based configurations to demonstrate sample deployment options.
• Cloud Build based end-to-end tests continuously run on each PR.
• A corpus of Wiki pages on how to use the components, connect them to external tools, and query the data-warehouse.