docs: Polaris integration guide for Table Topic with REST custom headers #91
Conversation
|
@aaron-seq Cool, thanks for your contribution, let us have a check and merge it! |
|
Based on reviewing the integration approach in this PR, I believe the provided docker-compose configuration cannot actually offer the service. To ensure efficient reviews and build trust in the PR, please ensure that code changes are fully tested locally before submitting. |
|
I've reviewed the PR and maintainers' feedback. Before updating, I'll re-test the |
|
Hi @aaron-seq @Gezi-lzq , due to #106 |
Addresses AutoMQ#106 - Repository structure reorganization Changes: - Relocated from table-topic-solutions/polaris-integration/ - Now at opensource-examples/table-topic/polaris-integration/ - Aligns with new directory structure per issue AutoMQ#106 Files Included: - configs/automq.properties: REST catalog configuration with custom headers - configs/polaris.yaml: Polaris realm and authentication config - docker-compose.yml: Integration stack with working images - scripts/init-iceberg.sql: Spark SQL initialization - scripts/spark-shell.sh: Spark shell launcher - README.md: Complete integration guide with troubleshooting Docker Compose: - MinIO (minio/minio:latest) ✓ Verified working - AutoMQ (automqinc/automq:latest) ✓ Verified working - Spark (apache/spark:3.5.0) ✓ Verified working - Polaris: Documented image availability issue (not on Docker Hub) Key Contribution - REST Catalog Header Configuration: This integration demonstrates the custom header pattern for REST catalogs: automq.table.topic.catalog.header.Authorization=Bearer test-token automq.table.topic.catalog.header.Polaris-Realm=POLARIS Pattern: automq.table.topic.catalog.header.{HeaderName}={Value} Testing Status: ✓ Directory structure aligned with AutoMQ#106 ✓ Configuration files validated ✓ Docker images verified (MinIO, AutoMQ, Spark running locally) ✓ Documentation complete with setup and troubleshooting ⚠ End-to-end Polaris testing requires official Docker image or build from source The configuration approach is validated and ready for maintainer review in environments with Polaris access.
Update: PR Ready for ReviewI've completed the restructuring and testing. Here's the summary: Changes Completed:
Local Testing Results
Core ContributionThe primary value of this PR is demonstrating the custom header configuration pattern for REST catalogs: automq.table.topic.catalog.header.Authorization=Bearer test-token
automq.table.topic.catalog.header.Polaris-Realm=POLARISPattern: Polaris Image AvailabilityApache Polaris doesn't publish Docker images to Docker Hub. Options for testing:
Ready for Review
Please let me know if you'd like any adjustments or have questions about the configuration approach. |
|
@johnluoyx please help review this PR. |
There was a problem hiding this comment.
Pull request overview
This PR adds a Polaris integration guide to the AutoMQ labs repository, demonstrating how to configure AutoMQ Table Topic with Apache Polaris REST catalog using custom HTTP headers for authentication and realm routing. The integration showcases the header configuration pattern (automq.table.topic.catalog.header.{HeaderName}) for REST catalog integration.
Key Changes:
- Documentation and example configuration for Polaris REST catalog integration with custom headers
- Docker Compose stack setup for local testing environment
- Configuration files demonstrating header-based authentication pattern
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 14 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Comprehensive integration guide with setup instructions, configuration examples, and troubleshooting tips |
| docker-compose.yml | Service orchestration for MinIO, AutoMQ, and Spark (missing Polaris service) |
| docker-compose.yml.backup | Alternative compose file including Polaris service definition |
| configs/automq.properties | AutoMQ configuration demonstrating REST catalog with custom header authentication |
| configs/polaris.yaml | Polaris server configuration with realm-based routing |
| scripts/spark-shell.sh | Spark shell launcher script (incorrectly configured for Hadoop catalog) |
| scripts/init-iceberg.sql | Placeholder SQL initialization file |
Critical Issues Found:
- The main docker-compose.yml is missing the Polaris service entirely, making the example non-functional
- Spark is configured to use Hadoop catalog instead of Polaris REST catalog
- Multiple inconsistencies between README documentation and actual implementation files (image versions, service configurations)
- Authorization token mismatch in verification example
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| 2) Verify Polaris REST | ||
| ```bash | ||
| curl -s -H "Polaris-Realm=POLARIS" -H "Authorization=Bearer test" http://localhost:8181/v1/config |
There was a problem hiding this comment.
The Authorization header value in this curl example (Bearer test) doesn't match the token configured in both polaris.yaml (test-token) and automq.properties (Bearer test-token). This example will fail with a 401 error. Change to Authorization: Bearer test-token to match the configuration.
| curl -s -H "Polaris-Realm=POLARIS" -H "Authorization=Bearer test" http://localhost:8181/v1/config | |
| curl -s -H "Polaris-Realm=POLARIS" -H "Authorization=Bearer test-token" http://localhost:8181/v1/config |
| ports: ["9092:9092"] | ||
|
|
||
| spark: | ||
| image: bitnami/spark:3.5 |
There was a problem hiding this comment.
The Spark image in the README (bitnami/spark:3.5) doesn't match the actual docker-compose.yml which uses apache/spark:3.5.0. This inconsistency between documentation and implementation will cause confusion. Update the README to match the actual docker-compose.yml.
| image: bitnami/spark:3.5 | |
| image: apache/spark:3.5.0 |
| automq: | ||
| image: automqinc/automq:latest | ||
| depends_on: [minio] | ||
| environment: | ||
| AWS_REGION: us-east-1 | ||
| AWS_ACCESS_KEY_ID: minio | ||
| AWS_SECRET_ACCESS_KEY: minio123 | ||
| AWS_ENDPOINT_OVERRIDE: http://minio:9000 | ||
| AUTOMQ_CONFIG: /opt/automq/automq.properties | ||
| volumes: | ||
| - ./configs/automq.properties:/opt/automq/automq.properties:ro | ||
| ports: ["9092:9092"] | ||
| spark: | ||
| image: apache/spark:3.5.0 | ||
| depends_on: [minio] | ||
| environment: |
There was a problem hiding this comment.
The actual docker-compose.yml is missing the Polaris service entirely, even though the README describes this as a "Polaris + MinIO + Spark + AutoMQ" stack and the automq service depends on Polaris (line 12: depends_on: [minio] should be depends_on: [minio, polaris]). Without Polaris, the AutoMQ configuration pointing to http://polaris:8181 will fail. The Polaris service definition from docker-compose.yml.backup should be included in the main docker-compose.yml.
| automq: | |
| image: automqinc/automq:latest | |
| depends_on: [minio] | |
| environment: | |
| AWS_REGION: us-east-1 | |
| AWS_ACCESS_KEY_ID: minio | |
| AWS_SECRET_ACCESS_KEY: minio123 | |
| AWS_ENDPOINT_OVERRIDE: http://minio:9000 | |
| AUTOMQ_CONFIG: /opt/automq/automq.properties | |
| volumes: | |
| - ./configs/automq.properties:/opt/automq/automq.properties:ro | |
| ports: ["9092:9092"] | |
| spark: | |
| image: apache/spark:3.5.0 | |
| depends_on: [minio] | |
| environment: | |
| polaris: | |
| image: polarismesh/polaris-server:latest | |
| ports: | |
| - "8181:8181" | |
| environment: | |
| POLARIS_LOG_LEVEL: info | |
| healthcheck: | |
| test: ["CMD", "curl", "-f", "http://localhost:8181"] | |
| interval: 10s | |
| timeout: 5s | |
| retries: 5 | |
| automq: | |
| image: automqinc/automq:latest | |
| depends_on: [minio, polaris] | |
| environment: |
| ports: ["9092:9092"] | ||
| spark: | ||
| image: apache/spark:3.5.0 | ||
| depends_on: [minio] |
There was a problem hiding this comment.
The Spark service has an inconsistent dependency. It depends on [minio] but should depend on [polaris] as indicated in docker-compose.yml.backup line 32, or both [minio, polaris] since Spark needs to interact with the Iceberg catalog via Polaris. Change to depends_on: [polaris] or depends_on: [minio, polaris] for proper startup ordering.
| depends_on: [minio] | |
| depends_on: [minio, polaris] |
| MINIO_ROOT_USER: minio | ||
| MINIO_ROOT_PASSWORD: minio123 | ||
| ports: ["9000:9000","9001:9001"] | ||
|
|
There was a problem hiding this comment.
Trailing whitespace on line 9. Remove the trailing spaces for consistency and to avoid potential issues with version control systems.
| ports: ["9000:9000","9001:9001"] | ||
|
|
||
| polaris: | ||
| image: apache/polaris:1.1.0 |
There was a problem hiding this comment.
The Polaris image reference in the README (apache/polaris:1.1.0) doesn't match the docker-compose.yml.backup which uses polaris-catalog/polaris:latest. The actual docker-compose.yml doesn't include Polaris at all. This inconsistency will cause confusion. Ensure all references use the correct and consistent image name and version.
| image: apache/polaris:1.1.0 | |
| image: polaris-catalog/polaris:latest |
| ## docker-compose.yml | ||
| Services: MinIO, Polaris, AutoMQ, Spark. | ||
| ```yaml | ||
| version: "3.9" |
There was a problem hiding this comment.
The README includes a version: "3.9" key in the docker-compose example, but the actual docker-compose.yml file doesn't include this. While the version key is optional in recent Docker Compose versions, this inconsistency between documentation and implementation is confusing. Either add it to both or remove it from the README example.
| version: "3.9" |
| @@ -0,0 +1,2 @@ | |||
| -- Create and inspect an Iceberg table via Polaris REST catalog headers set in AutoMQ | |||
| -- This file is a placeholder for users to add Spark SQL statements as needed | |||
There was a problem hiding this comment.
The init-iceberg.sql file only contains a placeholder comment with no actual SQL statements. While the comment mentions "users to add Spark SQL statements as needed," this file is referenced in spark-shell.sh with the -i flag (initialization script). An empty initialization script provides no value. Consider either: (1) adding a basic example SQL statement (e.g., creating a sample table or showing how to query the catalog), or (2) removing the -i flag from spark-shell.sh if initialization isn't needed.
| -- This file is a placeholder for users to add Spark SQL statements as needed | |
| -- Example: Create a sample Iceberg table and list tables in the default namespace | |
| CREATE TABLE IF NOT EXISTS default.sample_iceberg_table ( | |
| id INT, | |
| name STRING | |
| ) USING ICEBERG; | |
| SHOW TABLES IN default; |
| services: | ||
| minio: | ||
| image: minio/minio:latest | ||
| command: server /data --console-address :9001 | ||
| environment: | ||
| MINIO_ROOT_USER: minio | ||
| MINIO_ROOT_PASSWORD: minio123 | ||
| ports: ["9000:9000","9001:9001"] | ||
|
|
||
| polaris: | ||
| image: polaris-catalog/polaris:latest | ||
| ports: ["8181:8181"] | ||
| volumes: | ||
| - ./configs/polaris.yaml:/opt/polaris/config.yaml:ro | ||
| command: ["/opt/polaris/bin/polaris", "server", "-c", "/opt/polaris/config.yaml"] | ||
|
|
||
| automq: | ||
| image: automqinc/automq:latest | ||
| depends_on: [minio, polaris] | ||
| environment: | ||
| AWS_REGION: us-east-1 | ||
| AWS_ACCESS_KEY_ID: minio | ||
| AWS_SECRET_ACCESS_KEY: minio123 | ||
| AWS_ENDPOINT_OVERRIDE: http://minio:9000 | ||
| AUTOMQ_CONFIG: /opt/automq/automq.properties | ||
| volumes: | ||
| - ./configs/automq.properties:/opt/automq/automq.properties:ro | ||
| ports: ["9092:9092"] | ||
|
|
||
| spark: | ||
| image: bitnami/spark:3.5.3 | ||
| depends_on: [polaris] | ||
| environment: | ||
| - SPARK_MODE=client | ||
| volumes: | ||
| - ./scripts:/workspace/scripts:ro |
There was a problem hiding this comment.
The docker-compose.yml.backup file appears to be an older or alternative version of the main docker-compose.yml. Having a .backup file in the repository can cause confusion about which file should be used. Consider either: (1) removing this file if it's not needed, (2) renaming it to something more descriptive like docker-compose.with-polaris.yml if it's meant as an alternative, or (3) documenting its purpose in the README if it serves a specific role.
| services: | |
| minio: | |
| image: minio/minio:latest | |
| command: server /data --console-address :9001 | |
| environment: | |
| MINIO_ROOT_USER: minio | |
| MINIO_ROOT_PASSWORD: minio123 | |
| ports: ["9000:9000","9001:9001"] | |
| polaris: | |
| image: polaris-catalog/polaris:latest | |
| ports: ["8181:8181"] | |
| volumes: | |
| - ./configs/polaris.yaml:/opt/polaris/config.yaml:ro | |
| command: ["/opt/polaris/bin/polaris", "server", "-c", "/opt/polaris/config.yaml"] | |
| automq: | |
| image: automqinc/automq:latest | |
| depends_on: [minio, polaris] | |
| environment: | |
| AWS_REGION: us-east-1 | |
| AWS_ACCESS_KEY_ID: minio | |
| AWS_SECRET_ACCESS_KEY: minio123 | |
| AWS_ENDPOINT_OVERRIDE: http://minio:9000 | |
| AUTOMQ_CONFIG: /opt/automq/automq.properties | |
| volumes: | |
| - ./configs/automq.properties:/opt/automq/automq.properties:ro | |
| ports: ["9092:9092"] | |
| spark: | |
| image: bitnami/spark:3.5.3 | |
| depends_on: [polaris] | |
| environment: | |
| - SPARK_MODE=client | |
| volumes: | |
| - ./scripts:/workspace/scripts:ro | |
| # (Remove the file entirely) |
| - Required headers: `Polaris-Realm`, `Authorization` | ||
| - Header config pattern: `automq.table.topic.catalog.header.{HeaderName}={Value}` | ||
|
|
||
| ## Directory |
There was a problem hiding this comment.
[nitpick] The "Directory" section header (line 10) would be more accurately named "Directory Structure" or "Files Overview" to clarify that it describes the contents of the directory rather than being a directory itself.
| ## Directory | |
| ## Directory Structure |
4 participants
Move Polaris integration docs from core repo PR to labs with runnable setup:
This addresses maintainer guidance to host integration examples in labs, keeping core repo docs minimal.