Add Apache HBase Provider #8

Dimitrionian · 2026-01-13T08:49:08Z

Add Apache HBase Provider

This PR introduces a new Apache HBase Provider for Apache Airflow, enabling seamless integration with Apache HBase - a distributed, scalable, big data store built on Apache Hadoop.

Features

Table Operations: Create, delete, and manage HBase tables with column families
Data Operations: Insert, retrieve, scan, and batch operations on table data
Backup & Restore: Full and incremental backup operations with restore capabilities
Monitoring: Sensors for table existence, row counts, and column values
SSL/TLS Support: Secure connections with certificate validation
Kerberos Authentication: Enterprise authentication with keytab support
Connection Pooling: High-performance connection pooling with optimized batch processing
Performance Optimization: Configurable batch sizes and parallel execution

Basic HBase provider

Feature/ado 368

Feature/ado 336

ADO-334 Optimization (bulk insert and connection pools)

Build changes for python 3.10

Dimitrionian · 2026-01-16T09:49:15Z

airflow/providers/hbase/sensors/hbase.py

+            exists = bool(row_data)
+            self.log.info("Row %s in table %s exists: %s", self.row_key, self.table_name, exists)
+            return exists
+        except Exception as e:


Remove exception handling to forward exceptions higher to allow Airflow handling them and retry out of the box

Dimitrionian · 2026-01-16T09:49:21Z

airflow/providers/hbase/sensors/hbase.py

+            self.log.info("Table %s has %d rows, expected: %d", self.table_name, row_count,
+                          self.expected_count)
+            return row_count == self.expected_count
+        except Exception as e:


Remove exception handling to forward exceptions higher to allow Airflow handling them and retry out of the box

Dimitrionian · 2026-01-16T09:49:28Z

airflow/providers/hbase/sensors/hbase.py

+                self.column, self.row_key, self.expected_value, actual_value
+            )
+            return matches
+        except Exception as e:


Remove exception handling to forward exceptions higher to allow Airflow handling them and retry out of the box

Dimitrionian · 2026-01-16T08:30:46Z

dags/example_bash_operator.py

+}
+
+dag = DAG(
+    "example_hbase_backup_simple",


This is an old version from the development and testing process. The point is Airflow copies dags in /dags folder to operate. I will remove it

Are we need DAG-examples?
I don't like this default DAG-name without DAG in file system.
May be I don't right.

Oh, you right. I hide this comment.

Dimitrionian · 2026-01-16T09:49:41Z

airflow/providers/hbase/provider.yaml

+  - airflow.providers.hbase.example_dags.example_hbase
+  - airflow.providers.hbase.example_dags.example_hbase_ssl
+  - airflow.providers.hbase.example_dags.example_hbase_advanced
+  - airflow.providers.hbase.example_dags.example_hbase_backup_simple


This is an old version from the development and testing process. The point is Airflow copies dags in /dags folder to operate. I will remove it

Dimitrionian · 2026-01-16T09:50:01Z

airflow/providers/hbase/operators/hbase.py

+
+    def __init__(
+        self,
+        action: str,


Added Enum logic

Dimitrionian · 2026-01-16T09:50:12Z

airflow/providers/hbase/sensors/hbase.py

+            return False
+
+
+class HBaseRowCountSensor(BaseSensorOperator):


Warning added

Dimitrionian · 2026-01-16T09:51:07Z

airflow/providers/hbase/sensors/hbase.py

+                self.log.info("Row %s not found in table %s", self.row_key, self.table_name)
+                return False
+
+            actual_value = row_data.get(self.column.encode('utf-8'), b'').decode('utf-8')


Direct bytes comparison added, test added

Dimitrionian · 2026-01-16T09:59:50Z

airflow/providers/standard/provider.yaml

+name: Standard
+description: Empty standard provider
+state: ready
+source-date-epoch: 1734000000


It seems, I copied it from another provider but it doesn't matter: it's managed by a release manager

Dimitrionian · 2026-01-16T10:00:27Z

airflow/providers/hbase/provider.yaml

+    `Apache HBase <https://hbase.apache.org/>`__
+
+state: ready
+source-date-epoch: 1734000000


It seems, I copied it from another provider but it doesn't matter: it's managed by a release manager. Nonetheless, changed it to an actual date for the Hbase provider

…flow retry

Feature/hbase provider review

Add current date for Hbase provider

Dimitrionian added 30 commits December 15, 2025 21:01

ADO-330 Basic Hbase provider implementation

08c2c86

ADO-330 Advanced Hbase provider implementation

60cba34

ADO-330 Change branch to cache dependencies

e43cd78

ADO-330 Build from the current branch

c6da543

ADO-330 Fix google-re2 version

a3164f4

ADO-330 Fix uv version

e35c1e4

ADO-330 Add FERNET_KEY from env

df2a2a9

ADO-330 Fix params

1f9822f

ADO-330 Create backup/restore, intermediate results

4df3e03

Merge pull request #1 from arenadata/feature/ado-330

6527d17

Basic HBase provider

ADO-330 Create backup/restore, different approach

5fb9b15

ADO-330 Use SSH to make backups

7e354e9

ADO-330 Fix SSH backup logic, fix build problems

8e68d67

ADO-330 Fix SSH full backup logic

3809919

ADO-330 Resore SSH full backup logic

e00a236

ADO-330 Fix tests

5a82441

ADO-368 Update documentation

44f26a0

Merge pull request #3 from arenadata/feature/ado-368

269d5b4

Feature/ado 368

ADO-336 Basic Kerberos implementation

a7c244b

ADO-336 Remove redundant library use

8f2572e

ADO-336 Add example dag

8cb57d9

ADO-336 Add basic operations with Kerberos support

f9796c3

ADO-336 Refactor to use strategy

136ac50

ADO-336 Finish and test strategy

58285db

ADO-336 Fix all tests

b3e8a0d

ADO-336 Make example_hbase dag idempotent

a2c8378

ADO-336 Make example_hbase_advanced dag idempotent

1d0233a

ADO-336 Backup fix

a237472

ADO-336 Add the restore backup functionality dag

eb1e1d4

ADO-336 Fix dags examples

031dfa2

Dimitrionian added 10 commits December 26, 2025 16:07

ADO-336 Update the security docs

02b3013

Merge pull request #5 from arenadata/feature/ado-336

3df6ac1

Feature/ado 336

ADO-334 Use connection pools for hbase thrift operations

dab6266

ADO-334 Optimize bulk inserts

237e7fb

ADO-334 Make bulk inserts connection pooled

591392b

ADO-334 Update documentation

97d3bf8

Merge pull request #6 from arenadata/feature/ado-334

a25d09a

ADO-334 Optimization (bulk insert and connection pools)

Fix sphinxcontrib-serializinghtml version

1d975c6

Upgrade pip

da36222

Merge pull request #7 from arenadata/feature/build-3.10

9b6a47f

Build changes for python 3.10

Dimitrionian self-assigned this Jan 13, 2026

gaxeliy reviewed Jan 16, 2026

View reviewed changes

Dimitrionian added 18 commits January 16, 2026 13:23

Include hbase into prod list

daa5207

Remove Exception handling to forward an exception higher to allow Air…

373d333

…flow retry

Remove redundant dags

88f8541

Use enums instead of strings

69202f3

Add HBaseRowCountSensor warning

aba2d1f

Compare bytes directly to avoid UnicodeDecodeError on HBase binary data

c1a4a44

Merge pull request #9 from arenadata/feature/hbase-provider-review

075d0b2

Feature/hbase provider review

Add current date for Hbase provider

a31c495

Merge pull request #10 from arenadata/feature/hbase-provider-review

931de7f

Add current date for Hbase provider

Generalize a default connection name

4f79af7

Remove JAVA_HOME hardcode

02f3ccf

Remove hardcoded connection parameters

899cc34

Remove hasattr and delattr

8323b43

Remove redundant sss_conn_id parameter

ba1ce2d

Add if_exists parameter for better table handling

ad9c415

Add if_not_exists parameter for better table handling

e57e610

Fix docstrings

512f113

Make encoding arbitrary in HBaseScanOperator and HBaseBatchGetOperator

c2098e2

Add Apache HBase Provider #8

Are you sure you want to change the base?

Add Apache HBase Provider #8

Uh oh!

Conversation

Dimitrionian commented Jan 13, 2026

Add Apache HBase Provider

Features

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants