Skip to content

[Subtask] Add basic implementation of Hologres Catalog. #9913

@TimothyDing

Description

@TimothyDing

Describe the subtask

Implementation Status

The Hologres JDBC Catalog has been implemented and is located under catalogs-contrib/catalog-jdbc-hologres. The implementation includes the following completed features:

Completed: Core Catalog Infrastructure

  • HologresCatalog (HologresCatalog.java): Main catalog class extending JdbcCatalog, with shortName() returning "jdbc-hologres". Creates all necessary Hologres-specific converters and operations via factory methods (createExceptionConverter(), createJdbcTypeConverter(), createJdbcDatabaseOperations(), createJdbcTableOperations(), createJdbcColumnDefaultValueConverter()).
  • HologresCatalogOperations (HologresCatalogOperations.java): Extends JdbcCatalogOperations, handles PostgreSQL JDBC driver deregistration on close() to avoid memory leaks when the catalog is unloaded from IsolatedClassLoader.
  • HologresCatalogCapability (HologresCatalogCapability.java): Implements Capability, defines naming rules following PostgreSQL conventions (regex: ^[_a-zA-Z\p{L}/][\w\p{L}-$/=]{0,62}$) and reserves system schema names (pg_catalog, information_schema, hologres).

Completed: Schema (Database) Operations

  • HologresSchemaOperations (HologresSchemaOperations.java): Extends JdbcDatabaseOperations, implements schema CRUD operations.
    • Maps Hologres Database to JDBC Catalog, Hologres Schema to JDBC Schema.
    • Filters out 10 system schemas: pg_toast, pg_catalog, information_schema, hologres, hg_internal, hg_recyclebin, hologres_object_table, hologres_sample, hologres_streaming_mv, hologres_statistic.
    • Supports schema comments via COMMENT ON SCHEMA syntax.
    • Schema existence check via schemaExists() using DatabaseMetaData.getSchemas().

Completed: Table Operations

  • HologresTableOperations (HologresTableOperations.java): Extends JdbcTableOperations, implements RequireDatabaseOperation, provides full table lifecycle management.

    • Table Type Support: Lists and loads TABLE (regular tables and partition child tables) and PARTITIONED TABLE (partition parent tables) types. Excludes VIEW and FOREIGN TABLE from listing.

    • CREATE TABLE: Generates complete DDL with:

      • Column definitions with data type, nullable constraints, and default values
      • WITH clause for Hologres-specific properties (orientation, clustering_key, etc.)
      • HASH distribution via distribution_key in WITH clause
      • PRIMARY KEY constraints
      • Table and column comments via COMMENT ON TABLE/COLUMN statements
      • Single quote escaping in comments (e.g., 'It''s a test')
    • Generated Column Support: Detects UnparsedExpression default values and generates GENERATED ALWAYS AS (expr) STORED syntax instead of DEFAULT. Correctly places the STORED keyword before nullable constraints (NOT NULL / NULL). Distinguishes from auto-increment columns (GENERATED BY DEFAULT AS IDENTITY).

    • Partition Table Support:

      • Logical Partition Tables (Hologres V3.1+): Reads is_logical_partitioned_table and logical_partition_columns from hologres.hg_table_properties. Creates via LOGICAL PARTITION BY LIST(col1[, col2]) syntax. Supports 1–2 partition columns.
      • Physical Partition Tables: Reads partition information from pg_catalog.pg_partitioned_table (partition strategy and attribute numbers) and pg_catalog.pg_attribute (attribute number to column name resolution). Creates via PARTITION BY LIST(column) syntax. Supports exactly 1 partition column.
    • ALTER TABLE: Supports the following change types:

      • UpdateComment: Update table comment with Gravitino ID preservation
      • AddColumn: Add column with type and comment only. Setting NOT NULL constraint, default value, and auto-increment are not supported via ALTER TABLE (throws IllegalArgumentException).
      • RenameColumn: Rename column via RENAME COLUMN
      • UpdateColumnDefaultValue: Throws IllegalArgumentException (Hologres does not support altering column default value via ALTER TABLE)
      • UpdateColumnType: Throws IllegalArgumentException (Hologres does not support altering column type via ALTER TABLE)
      • UpdateColumnComment: Update column comment via COMMENT ON COLUMN
      • DeleteColumn: Drop column with IF EXISTS support
      • UpdateColumnNullability: Throws IllegalArgumentException (Hologres does not support altering column nullability via ALTER TABLE)
      • AddIndex: Throws IllegalArgumentException (Hologres does not support adding index via ALTER TABLE)
      • DeleteIndex: Throws IllegalArgumentException (Hologres does not support deleting index via ALTER TABLE)
      • UpdateColumnAutoIncrement: Throws IllegalArgumentException (Hologres does not support altering auto-increment via ALTER TABLE)
      • UpdateColumnPosition: Throws IllegalArgumentException (Hologres does not support column reordering)
      • SetProperty / RemoveProperty: Throws IllegalArgumentException (not yet supported)
    • Table Properties: Reads from hologres.hg_table_properties system table. Handles JDBC-to-DDL property key conversion (e.g., binlog.levelbinlog_level, binlog.ttlbinlog_ttl). Filters internal properties via isUserRelevantProperty() and only exposes user-relevant ones. During CREATE TABLE, filters distribution_key, is_logical_partitioned_table, and primary_key from the WITH clause to avoid duplication.

    • Distribution: Reads distribution key from hologres.hg_table_properties and returns Distributions.hash(). Writes distribution key via the WITH clause. Validates that only HASH distribution strategy is used and at least one column is specified.

    • Auto-increment: Not supported. Creating auto-increment columns (e.g., GENERATED BY DEFAULT AS IDENTITY) is rejected with IllegalArgumentException in both CREATE TABLE and ALTER TABLE ADD COLUMN. Hologres SERIAL/auto-increment tables that already exist can still be loaded and displayed, but cannot be created through Gravitino.

Completed: Type Conversion

  • HologresTypeConverter (HologresTypeConverter.java): Bidirectional type mapping between Hologres and Gravitino types (185 lines).

    • Handles TIMESTAMP/TIMESTAMPTZ without precision suffix (Hologres does not support precision syntax like timestamptz(6)).
    • Supports array types via _prefix to [] suffix conversion (e.g., _int4int4[]).
    • Maps unsupported types (json, jsonb, uuid, etc.) to ExternalType.
    • Array elements must be non-nullable (Hologres limitation).
    • Multidimensional arrays are not supported.
  • HologresColumnDefaultValueConverter (HologresColumnDefaultValueConverter.java): Handles PostgreSQL-style default value casting (133 lines).

    • toGravitino(): Parses literal default values with type cast patterns (e.g., '2024-01-15'::date), handles NULL::type cast expressions, returns CURRENT_TIMESTAMP for the corresponding PostgreSQL value, and falls back to UnparsedExpression for complex expressions (e.g., generated column expressions, nextval('seq')).
    • fromGravitino(): Overridden to support converting UnparsedExpression back to SQL strings (for generated column round-trip), delegates to super for Literal and DEFAULT_VALUE_NOT_SET.
  • HologresExceptionConverter (HologresExceptionConverter.java): Maps PostgreSQL SQLSTATE error codes to Gravitino exceptions (85 lines).

    • 42P04/42P06SchemaAlreadyExistsException
    • 42P07TableAlreadyExistsException
    • 3D000/3F000NoSuchSchemaException
    • 42P01NoSuchTableException
    • 08xxxConnectionFailedException
    • Password authentication failure → ConnectionFailedException

Completed: Configuration and Build

  • Build: build.gradle.kts (112 lines) with dependencies on catalog-jdbc-common, api, common, core, PostgreSQL driver for testing, testcontainers support, Guava, Commons Lang3/Collections4, and Log4j.
  • Config: jdbc-hologres.conf with JDBC connection configuration template.
  • Service Registration: META-INF/services/org.apache.gravitino.CatalogProvider for catalog auto-discovery.
  • Distribution: Build tasks (copyCatalogLibs, copyCatalogConfig) to package catalog into distribution/package/catalogs/jdbc-hologres/.

Completed: Unit Tests

5 test classes with comprehensive coverage:

  • TestHologresTypeConverter: Covers all supported type conversions including boolean, integer types, float types, date/time types, decimal, varchar, char, text, binary, array types, external types, and error cases for nullable arrays and multidimensional arrays.

  • TestHologresColumnDefaultValueConverter: Covers toGravitino() for all literal types (boolean, short, integer, long, float, double, decimal, varchar, text, bpchar, date, time, timestamp), expression fallback to UnparsedExpression, NULL::type handling, CURRENT_TIMESTAMP, null default value (nullable vs non-nullable), unknown type fallback, and fromGravitino() for UnparsedExpression, Literal, and DEFAULT_VALUE_NOT_SET.

  • TestHologresExceptionConverter: Covers all SQLSTATE error code mappings (42P04, 42P06, 42P07, 3D000, 3F000, 42P01, 08xxx), null SQL state handling, and password authentication failure detection.

  • TestHologresCatalogCapability: Covers naming validation patterns (valid names with letters, underscores, digits, hyphens, dollar signs, slashes, Unicode), invalid names (starting with digit, empty, too long), max length validation, and reserved word enforcement for SCHEMA scope.

  • TestHologresTableOperations (830 lines, 35+ test cases): Covers:

    • CREATE TABLE SQL generation: Basic, with comments, column comments, distribution, properties, property filtering for distribution_key/is_logical_partitioned_table/primary_key, physical partition, logical partition (1 and 2 columns), auto-increment rejection, nullable columns, default values, multiple columns, single quote escaping in comments, full-featured table, generated columns (non-nullable and nullable).
    • RENAME TABLE, DROP TABLE, PURGE TABLE (unsupported exception).
    • Partitioning SQL: Physical, logical (single/two columns), rejects non-list transform, rejects multiple columns for physical, rejects >2 columns for logical, rejects empty partitions, rejects multiple transforms, rejects nested field names.
    • Index field string generation: Single column, multiple columns, rejects nested column.
    • Datetime precision calculation: Timestamp, timestamptz, time, timetz, non-datetime type, negative scale.

    Note: ALTER TABLE operations UpdateColumnDefaultValue, UpdateColumnType, UpdateColumnNullability, AddIndex, DeleteIndex, and UpdateColumnAutoIncrement are all rejected with IllegalArgumentException in generateAlterTableSql(). CREATE TABLE with indexes and constraints is still supported.

Parent issue

#9886

Metadata

Metadata

Assignees

No one assigned

    Labels

    subtaskSubtasks of umbrella issue

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions