feat(ingestion/hive-metastore): add upstream lineage to hive-metastore #15435

acrylJonny · 2025-11-27T16:02:46Z

Add Storage Lineage to Hive/Hive Metastore + Code Refactoring

Summary

Adds storage lineage support to Hive and Hive Metastore connectors, enabling lineage tracking between Hive tables and their underlying storage locations (S3, Azure, GCS, HDFS, DBFS). Also refactors Hive sources into a clean directory structure.

Key Changes

Storage Lineage (Opt-in Feature)

New configuration options (disabled by default):

emit_storage_lineage: Enable storage lineage extraction
hive_storage_lineage_direction: Set direction (upstream or downstream)
include_column_lineage: Enable column-level lineage
storage_platform_instance: Platform instance for storage URNs

Supported platforms: S3, Azure (ADLS/ABFS), GCS, HDFS, DBFS, local files

Code Refactoring

Moved hive.py → hive/hive_source.py
Moved hive_metastore.py → hive/hive_metastore_source.py
Created hive/storage_lineage.py with shared logic
Extracted HiveStorageLineageConfigMixin to eliminate duplication
Updated setup.py entry points to use fully qualified paths

Code Quality Improvements

Converted HiveStorageLineageConfig to Pydantic model
Created LineageDirection and StoragePlatform StrEnums for type safety
Fixed unsafe exception handling in get_db_schema (now raises ValueError for invalid input)
Improved error reporting in get_workunits_internal (specific exceptions + proper logging)
Removed 50+ redundant comments

PR Checks

The PR conforms to DataHub's Contributing Guideline (particularly PR Title Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

codecov · 2025-11-27T16:05:19Z

Codecov Report

❌ Patch coverage is 83.97933% with 62 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...c/datahub/ingestion/source/sql/hive/hive_source.py	79.06%	36 Missing ⚠️
...tahub/ingestion/source/sql/hive/storage_lineage.py	90.41%	16 Missing ⚠️
...ingestion/source/sql/hive/hive_metastore_source.py	77.27%	10 Missing ⚠️

📢 Thoughts on this report? Let us know!

sgomezvillamor · 2025-12-03T06:06:39Z

metadata-ingestion/src/datahub/ingestion/source/sql/hive/hive_metastore_source.py

        description="Simplify v2 field paths to v1 by default. If the schema has Union or Array types, still falls back to v2",
    )

+    emit_storage_lineage: bool = Field(


do we usually call this emit_... or include_...?

sgomezvillamor · 2025-12-03T06:09:06Z

metadata-ingestion/src/datahub/ingestion/source/sql/hive/hive_metastore_source.py

+        default=False,
+        description="Whether to emit storage-to-Hive lineage",
+    )
+    hive_storage_lineage_direction: str = Field(


this ability to choose direction, do we have something similar in other sources?

anyway, it could be a Literal https://docs.pydantic.dev/1.10/usage/types/#literal-type

We also have this already in this in the main hive source (i.e. not the hive metastore source)

Therefore adding it here also for consistency. One of the main reasons for this is because Spark often can show lineage just into the files rather than the metastore depending on how the table is updated.

aikido-pr-checks · 2025-12-03T17:59:14Z

metadata-ingestion/src/datahub/ingestion/source/sql/hive/hive_source.py

+        if self._COMPLEX_TYPE.match(fields[0].nativeDataType) and isinstance(
+            fields[0].type.type, NullTypeClass
+        ):
+            assert len(fields) == 1


Dangerous use of assert - low severity
When running Python in production in optimized mode, assert calls are not executed. This mode is enabled by setting the PYTHONOPTIMIZE command line flag. Optimized mode is usually ON in production. Any safety check done using assert will not be executed.

Remediation: Raise an exception instead of using assert.
^{View details in Aikido Security}

sgomezvillamor · 2025-12-04T14:57:21Z

metadata-ingestion/docs/sources/hive-metastore/hive-metastore_pre.md

Great docs!

Still a new user may wonder: which one hive or hive-metastore should I use? is hive going to be deprecated? is there feature parity or what's the feature that one has that the other misses?

And for existing users of hive: what's the plan? is it going to be deprecated eventually? are new features going to be implemented in the hive one too?

sgomezvillamor · 2025-12-04T14:59:24Z

metadata-ingestion/docs/sources/hive/hive_pre.md

+This creates URNs like:
+
+```
+urn:li:dataset:(urn:li:dataPlatform:hive,database.table,prod-hive)


both hive and hive-metastore generate URNs with hive platform in the URN, right?
so if a user moves from hive to hive-metastore that won't be a breaking change in the identiyy of the assets, right?

That is correct. You can move between the two and the URNs generated ultimately are the same but using different approaches (metastore db vs thrift). This hasn't changed.

sgomezvillamor · 2025-12-04T15:08:44Z

metadata-ingestion/src/datahub/ingestion/source/sql/hive/hive_metastore_source.py

-            # add view properties
-            properties: Dict[str, str] = {
-                "is_view": "True",
-            }
+            properties: Dict[str, str] = {"is_view": "True"}


this is not new behaviour

anyway, why not using SubTypes aspect here to track this as a view?

happy to make this a view subtype - just didn't want to introduce this as it is a change in the connector. Happy to do so if you think that's wise.

metadata-ingestion/src/datahub/ingestion/source/sql/hive/hive_source.py

sgomezvillamor · 2025-12-04T15:17:56Z

metadata-ingestion/src/datahub/ingestion/source/sql/hive/hive_source.py

+                    "type": coltype,
+                    "nullable": True,
+                    "default": None,
+                    "full_type": orig_col_type,  # pass it through


so full_type, this is what we really miss in the original non-patched method?

what would be the impact of missing int?
patch is a heavy tech debt, so wondering if the value worths it

This is pre-existing -

datahub/metadata-ingestion/src/datahub/ingestion/source/sql/hive.py

Lines 605 to 612 in ddd9321

result.append(

{

"name": col_name,

"type": coltype,

"nullable": True,

"default": None,

"full_type": orig_col_type, # pass it through

"comment": _comment,

so this patch is not new? is that what you mean?

sgomezvillamor · 2025-12-04T15:19:34Z

metadata-ingestion/src/datahub/ingestion/source/sql/hive/hive_source.py

+HiveDialect.get_view_names = get_view_names_patched
+HiveDialect.get_view_definition = get_view_definition_patched


HiveDialect, this comes from our fork, right? https://github.com/acryldata/PyHive
May we just fix there?

sgomezvillamor · 2025-12-04T15:23:25Z

metadata-ingestion/src/datahub/ingestion/source/sql/hive/storage_lineage.py

+        except (ValueError, TypeError, AttributeError) as e:
+            logger.warning(
+                f"Failed to create storage dataset MCPs for {storage_location}: {e}",
+                exc_info=True,
+            )
+            return


this exception swallowing pattern, is that what we usually do in other sources when we fail emission?

sgomezvillamor · 2025-12-04T15:27:38Z

codecoverage report shows little coverage in failure/exception scenarios, we could iimprove a little bit there, mainly in hive_source.py

also, concerning the lack of coverage for dbapi_get_columns_patched

acrylJonny · 2025-12-04T17:32:32Z

codecoverage report shows little coverage in failure/exception scenarios, we could iimprove a little bit there, mainly in hive_source.py

also, concerning the lack of coverage for dbapi_get_columns_patched

I'll add these today so that we can get this signed off.

feat(ingestion/hive-metastore): add upstream lineage to hive-metastore

2a3c29a

acrylJonny marked this pull request as draft November 27, 2025 16:02

github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Nov 27, 2025

github-actions bot deployed to datahub-wheels (Preview) November 27, 2025 16:04 View deployment

vercel bot deployed to Preview November 27, 2025 16:21 View deployment

acrylJonny and others added 2 commits November 27, 2025 17:11

docs fixes

102a72d

Merge branch 'master' into hive-metastore-upstreams

3300fca

github-actions bot deployed to datahub-wheels (Preview) November 27, 2025 17:14 View deployment

acrylJonny marked this pull request as ready for review November 27, 2025 17:15

datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Nov 27, 2025

vercel bot deployed to Preview November 27, 2025 17:31 View deployment

updating golden files

5284812

github-actions bot deployed to datahub-wheels (Preview) November 28, 2025 15:14 View deployment

vercel bot deployed to Preview November 28, 2025 15:31 View deployment

sgomezvillamor reviewed Dec 3, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Dec 3, 2025

addressing comments

f843ba7

datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Dec 3, 2025

github-actions bot deployed to datahub-wheels (Preview) December 3, 2025 12:42 View deployment

vercel bot deployed to Preview December 3, 2025 12:59 View deployment

fixes and tests

db424b0

github-actions bot deployed to datahub-wheels (Preview) December 3, 2025 13:37 View deployment

vercel bot deployed to Preview December 3, 2025 13:53 View deployment

further fixes

402a842

github-actions bot deployed to datahub-wheels (Preview) December 3, 2025 15:16 View deployment

vercel bot deployed to Preview December 3, 2025 15:33 View deployment

vercel bot deployed to Preview December 3, 2025 16:40 View deployment

adding docs

99450ee

aikido-pr-checks bot reviewed Dec 3, 2025

View reviewed changes

github-actions bot deployed to datahub-wheels (Preview) December 3, 2025 18:01 View deployment

vercel bot had a problem deploying to Preview December 3, 2025 18:14 Failure

further feedback addressed

1fc5519

github-actions bot deployed to datahub-wheels (Preview) December 3, 2025 18:56 View deployment

vercel bot had a problem deploying to Preview December 3, 2025 19:08 Failure

Update presto_pre.md

b8aee09

github-actions bot deployed to datahub-wheels (Preview) December 3, 2025 19:34 View deployment

vercel bot deployed to Preview December 3, 2025 19:50 View deployment

acrylJonny and others added 2 commits December 3, 2025 19:56

Update hive_metastore_source.py

aa559a5

Merge branch 'master' into hive-metastore-upstreams

040471b

github-actions bot deployed to datahub-wheels (Preview) December 3, 2025 19:59 View deployment

vercel bot deployed to Preview December 3, 2025 20:16 View deployment

sgomezvillamor reviewed Dec 4, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Dec 4, 2025

sgomezvillamor reviewed Dec 4, 2025

View reviewed changes

metadata-ingestion/src/datahub/ingestion/source/sql/hive/hive_source.py Show resolved Hide resolved

sgomezvillamor reviewed Dec 4, 2025

View reviewed changes

	result.append(
	{
	"name": col_name,
	"type": coltype,
	"nullable": True,
	"default": None,
	"full_type": orig_col_type, # pass it through
	"comment": _comment,

		HiveDialect.get_view_names = get_view_names_patched
		HiveDialect.get_view_definition = get_view_definition_patched

feat(ingestion/hive-metastore): add upstream lineage to hive-metastore #15435

Are you sure you want to change the base?

feat(ingestion/hive-metastore): add upstream lineage to hive-metastore #15435

Conversation

acrylJonny commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add Storage Lineage to Hive/Hive Metastore + Code Refactoring

Summary

Key Changes

Storage Lineage (Opt-in Feature)

Code Refactoring

Code Quality Improvements

PR Checks

Uh oh!

codecov bot commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aikido-pr-checks bot Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

acrylJonny Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sgomezvillamor commented Dec 4, 2025

Uh oh!

acrylJonny commented Dec 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

acrylJonny commented Nov 27, 2025 •

edited

Loading

codecov bot commented Nov 27, 2025 •

edited

Loading

aikido-pr-checks bot Dec 3, 2025 •

edited

Loading

acrylJonny Dec 4, 2025 •

edited

Loading