diff --git a/connectors/dashboard/domo-dashboard/index.mdx b/connectors/dashboard/domo-dashboard/index.mdx index 7bfc3a8a..0ff152dd 100644 --- a/connectors/dashboard/domo-dashboard/index.mdx +++ b/connectors/dashboard/domo-dashboard/index.mdx @@ -27,7 +27,7 @@ For metadata ingestion, make sure to add at least `data` scopes to the clientId For questions related to scopes, click [here](https://developer.domo.com/portal/1845fc11bbe5d-api-authentication). ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/dashboard/lightdash/index.mdx b/connectors/dashboard/lightdash/index.mdx index 88e345d0..47ef12bf 100644 --- a/connectors/dashboard/lightdash/index.mdx +++ b/connectors/dashboard/lightdash/index.mdx @@ -28,7 +28,7 @@ Configure and schedule Lightdash metadata and profiler workflows from the Collat To integrate Lightdash, ensure you are using OpenMetadata version 1.2.x or higher. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/dashboard/looker/index.mdx b/connectors/dashboard/looker/index.mdx index ccc10f2c..fb68cace 100644 --- a/connectors/dashboard/looker/index.mdx +++ b/connectors/dashboard/looker/index.mdx @@ -44,7 +44,7 @@ We do not yet support liquid variables. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/dashboard/metabase/index.mdx b/connectors/dashboard/metabase/index.mdx index aaf3b007..9011a22f 100644 --- a/connectors/dashboard/metabase/index.mdx +++ b/connectors/dashboard/metabase/index.mdx @@ -28,7 +28,7 @@ The service account must have view access to all dashboards and charts that need **Note:** We have tested Metabase with Versions `0.42.4` and `0.43.4`. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/dashboard/microstrategy/index.mdx b/connectors/dashboard/microstrategy/index.mdx index 7dfeb7bc..2a481f74 100644 --- a/connectors/dashboard/microstrategy/index.mdx +++ b/connectors/dashboard/microstrategy/index.mdx @@ -34,7 +34,7 @@ However, if the user still cannot access the APIs, the following should be check - Browse Repository: Object navigation within projects/folders. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/dashboard/mode/index.mdx b/connectors/dashboard/mode/index.mdx index ba07b403..d7178e86 100644 --- a/connectors/dashboard/mode/index.mdx +++ b/connectors/dashboard/mode/index.mdx @@ -27,7 +27,7 @@ Configure and schedule Mode metadata and profiler workflows from the Collate UI: OpenMetadata relies on Mode's API, which is exclusive to members of the Mode Business Workspace. This means that only resources that belong to a Mode Business Workspace can be accessed via the API. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/dashboard/powerbi/index.mdx b/connectors/dashboard/powerbi/index.mdx index 7bd30f19..3b17ab10 100644 --- a/connectors/dashboard/powerbi/index.mdx +++ b/connectors/dashboard/powerbi/index.mdx @@ -76,7 +76,7 @@ Create new workspaces in PowerBI by following the document [here](https://docs.m For reference here is a [thread](https://community.powerbi.com/t5/Service/Error-while-executing-Get-dataset-call-quot-API-is-not/m-p/912360#M85711) referring to the same ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/dashboard/powerbireportserver/index.mdx b/connectors/dashboard/powerbireportserver/index.mdx index e2866fd9..e00567e2 100644 --- a/connectors/dashboard/powerbireportserver/index.mdx +++ b/connectors/dashboard/powerbireportserver/index.mdx @@ -27,7 +27,7 @@ Configure and schedule PowerBI Report Server metadata from CLI: The PowerBI Report Server should be accessible from the ingestion environment. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/dashboard/qlikcloud/api_keys.mdx b/connectors/dashboard/qlikcloud/api_keys.mdx index a24a33a7..fe81d418 100644 --- a/connectors/dashboard/qlikcloud/api_keys.mdx +++ b/connectors/dashboard/qlikcloud/api_keys.mdx @@ -12,13 +12,13 @@ OpenMetadata Uses [REST APIs](https://qlik.dev/apis/) to communicate with Qlik C In this document we will explain how you can generate this token so that OpenMetadata can communicate with Qlik Cloud. -# Step 1: Open Qlik Cloud Management Console (QMC) +## Step 1: Open Qlik Cloud Management Console (QMC) Open your Qlik Cloud Management Console (QMC) and navigate to API Keys section. Navigate to API Keys in QMC -# Step 2: Provide Name and Generate API Key +## Step 2: Provide Name and Generate API Key 1. Provide name for the API key you will generate. diff --git a/connectors/dashboard/qlikcloud/index.mdx b/connectors/dashboard/qlikcloud/index.mdx index e25d67e6..41ce8187 100644 --- a/connectors/dashboard/qlikcloud/index.mdx +++ b/connectors/dashboard/qlikcloud/index.mdx @@ -29,7 +29,7 @@ Configure and schedule QlikCloud metadata and profiler workflows from the Collat ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/dashboard/qliksense/certificates.mdx b/connectors/dashboard/qliksense/certificates.mdx index 26370048..675cf501 100644 --- a/connectors/dashboard/qliksense/certificates.mdx +++ b/connectors/dashboard/qliksense/certificates.mdx @@ -13,13 +13,13 @@ OpenMetadata Uses [Qlik Engine APIs](https://help.qlik.com/en-US/sense-developer In this document we will explain how you can generate these certificates so that OpenMetadata can communicate with Qlik Sense. -# Step 1: Open Qlik Management Console (QMC) +## Step 1: Open Qlik Management Console (QMC) Open your Qlik Management Console (QMC) and navigate to certificates section. Navigate to certificates in QMC -# Step 2: Provide Details and Export Certificates +## Step 2: Provide Details and Export Certificates 1. In the Machine name box, type the full computer name of the computer that you are creating the certificates for: MYMACHINE.mydomain.com or the IP address. @@ -35,7 +35,7 @@ Open your Qlik Management Console (QMC) and navigate to certificates section. Provide Certificate Details -# Step 3: Locate the certificates +## Step 3: Locate the certificates Once you have exported the certificates you can see the location of exported certificates just below the certificate details page. When you navigate to that location you will find the `root.pem`, `client.pem` & `client_key.pem` certificates which will be used by OpenMetadata. diff --git a/connectors/dashboard/qliksense/index.mdx b/connectors/dashboard/qliksense/index.mdx index b31bd315..ca55d1a9 100644 --- a/connectors/dashboard/qliksense/index.mdx +++ b/connectors/dashboard/qliksense/index.mdx @@ -30,7 +30,7 @@ Configure and schedule Metabase metadata and profiler workflows from the Collate ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/dashboard/quicksight/index.mdx b/connectors/dashboard/quicksight/index.mdx index 52840b53..b93bc054 100644 --- a/connectors/dashboard/quicksight/index.mdx +++ b/connectors/dashboard/quicksight/index.mdx @@ -64,7 +64,7 @@ Here is how to add Permissions to an IAM user. ``` ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/dashboard/redash/index.mdx b/connectors/dashboard/redash/index.mdx index fc7a4467..ed0a0178 100644 --- a/connectors/dashboard/redash/index.mdx +++ b/connectors/dashboard/redash/index.mdx @@ -29,7 +29,7 @@ we use in the configuration to ingest data must have enough permissions to view permissions, please visit Redash documentation [here](https://redash.io/help/user-guide/users/permissions-groups). ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/dashboard/sigma/index.mdx b/connectors/dashboard/sigma/index.mdx index d7844a6f..a42d886b 100644 --- a/connectors/dashboard/sigma/index.mdx +++ b/connectors/dashboard/sigma/index.mdx @@ -27,7 +27,7 @@ Configure and schedule Sigma metadata and profiler workflows from the Collate UI OpenMetadata relies on Sigma's REST API. To know more you can read the [Sigma API Get Started docs](https://help.sigmacomputing.com/reference/get-started-sigma-api#about-the-api). To [generate API client credentials](https://help.sigmacomputing.com/reference/generate-client-credentials#user-requirements), you must be assigned the Admin account type. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/dashboard/superset/index.mdx b/connectors/dashboard/superset/index.mdx index 61b65d03..82c995d8 100644 --- a/connectors/dashboard/superset/index.mdx +++ b/connectors/dashboard/superset/index.mdx @@ -31,7 +31,7 @@ The ingestion also works with Superset 2.0.0 🎉 **Database Connection**: To extract metadata from Superset via MySQL or Postgres database, database user must have at least `SELECT` privilege on `dashboards` & `slices` tables within superset schema. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/dashboard/tableau/index.mdx b/connectors/dashboard/tableau/index.mdx index 5c5e7037..f5257fb6 100644 --- a/connectors/dashboard/tableau/index.mdx +++ b/connectors/dashboard/tableau/index.mdx @@ -71,7 +71,7 @@ This mapping ensures that: - Chrome Extension compatibility is maintained ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/cassandra/hybrid-runner.mdx b/connectors/database/cassandra/hybrid-runner.mdx index fbf1dfc7..b33089a7 100644 --- a/connectors/database/cassandra/hybrid-runner.mdx +++ b/connectors/database/cassandra/hybrid-runner.mdx @@ -29,7 +29,7 @@ To extract metadata using the Cassandra connector, ensure the user in the connec - Schema Operations: Access to list and describe keyspaces and tables. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/cassandra/index.mdx b/connectors/database/cassandra/index.mdx index edd210fb..ead5781f 100644 --- a/connectors/database/cassandra/index.mdx +++ b/connectors/database/cassandra/index.mdx @@ -29,7 +29,7 @@ To extract metadata using the Cassandra connector, ensure the user in the connec - Schema Operations: Access to list and describe keyspaces and tables. ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Username to connect to Cassandra. This user must have the necessary permissions to perform metadata extraction and table queries. diff --git a/connectors/database/clickhouse/hybrid-runner.mdx b/connectors/database/clickhouse/hybrid-runner.mdx index 4287f06b..474a932d 100644 --- a/connectors/database/clickhouse/hybrid-runner.mdx +++ b/connectors/database/clickhouse/hybrid-runner.mdx @@ -50,7 +50,7 @@ Executing the profiler workflow or data quality tests, will require the user to For the usage and lineage workflow, the user will need `SELECT` privilege. You can find more information on the usage workflow [here](/how-to-guides/guide-for-data-users/ingestion/workflows/usage) and the lineage workflow [here](/how-to-guides/guide-for-data-users/ingestion/workflows/lineage). ## Metadata Ingestion -# Connection Options +## Connection Options diff --git a/connectors/database/cockroach/hybrid-runner.mdx b/connectors/database/cockroach/hybrid-runner.mdx index 31567825..ef7d6273 100644 --- a/connectors/database/cockroach/hybrid-runner.mdx +++ b/connectors/database/cockroach/hybrid-runner.mdx @@ -33,7 +33,7 @@ To use the Cockroach connector with the Hybrid Ingestion Runner, ensure: - The Hybrid Ingestion Runner is installed and configured, and can reach both CockroachDB and the Collate SaaS control plane. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/cockroach/index.mdx b/connectors/database/cockroach/index.mdx index 46f252b7..e2240f39 100644 --- a/connectors/database/cockroach/index.mdx +++ b/connectors/database/cockroach/index.mdx @@ -27,7 +27,7 @@ Configure and schedule Cockroach metadata workflows from the Collate UI: ## Requirements ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Specify the User to connect to Cockroach. It should have enough privileges to read all the metadata. diff --git a/connectors/database/couchbase/hybrid-runner.mdx b/connectors/database/couchbase/hybrid-runner.mdx index 193cc4a0..6fd8e432 100644 --- a/connectors/database/couchbase/hybrid-runner.mdx +++ b/connectors/database/couchbase/hybrid-runner.mdx @@ -24,7 +24,7 @@ Configure and schedule Couchbase metadata workflows from the Collate UI: - [Troubleshooting](/connectors/database/couchbase/troubleshooting) ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/couchbase/index.mdx b/connectors/database/couchbase/index.mdx index 2608f275..3111ba87 100644 --- a/connectors/database/couchbase/index.mdx +++ b/connectors/database/couchbase/index.mdx @@ -24,7 +24,7 @@ Configure and schedule Couchbase metadata workflows from the Collate UI: - [Troubleshooting](/connectors/database/couchbase/troubleshooting) ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Username to connect to Couchbase. diff --git a/connectors/database/databricks/hybrid-runner.mdx b/connectors/database/databricks/hybrid-runner.mdx index 1ca4efdf..04c8c927 100644 --- a/connectors/database/databricks/hybrid-runner.mdx +++ b/connectors/database/databricks/hybrid-runner.mdx @@ -65,7 +65,7 @@ Adjust <user>, <catalog_name>, <schema_name>, and <table_na If you are using Unity Catalog in Databricks, then checkout the [Unity Catalog](/connectors/database/unity-catalog) connector. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/databricks/index.mdx b/connectors/database/databricks/index.mdx index 46a9ff9f..33b2da9c 100644 --- a/connectors/database/databricks/index.mdx +++ b/connectors/database/databricks/index.mdx @@ -65,7 +65,7 @@ Adjust <user>, <catalog_name>, <schema_name>, and <table_na If you are using Unity Catalog in Databricks, then checkout the [Unity Catalog](/connectors/database/unity-catalog) connector. ## Metadata Ingestion -# Connection Details +## Connection Details - **Host and Port**: Enter the fully qualified hostname and port number for your Databricks deployment in the Host and Port field. diff --git a/connectors/database/db2/hybrid-runner.mdx b/connectors/database/db2/hybrid-runner.mdx index 8bc73c94..84f0163f 100644 --- a/connectors/database/db2/hybrid-runner.mdx +++ b/connectors/database/db2/hybrid-runner.mdx @@ -115,7 +115,7 @@ GRANT SELECT ON SYSIBM.SYSSEQUENCES TO USER_NAME; Executing the profiler workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/how-to-guides/data-quality-observability/quality). ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/db2/index.mdx b/connectors/database/db2/index.mdx index d754eb9f..8a5ba7e3 100644 --- a/connectors/database/db2/index.mdx +++ b/connectors/database/db2/index.mdx @@ -115,7 +115,7 @@ GRANT SELECT ON SYSIBM.SYSSEQUENCES TO USER_NAME; Executing the profiler workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/how-to-guides/data-quality-observability/quality). ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Specify the User to connect to DB2. It should have enough privileges to read all the metadata. diff --git a/connectors/database/dbt/configure-dbt-workflow.mdx b/connectors/database/dbt/configure-dbt-workflow.mdx index 304c9fd4..9f79fad3 100644 --- a/connectors/database/dbt/configure-dbt-workflow.mdx +++ b/connectors/database/dbt/configure-dbt-workflow.mdx @@ -13,7 +13,7 @@ name="dbt" stage="PROD" availableFeatures={["Metadata", "Queries", "Lineage", "Tags", "Tiers", "Domains", "Custom Properties", "Glossary", "Owners", "Descriptions", "Tests", "Exposures"]} /> -# Configure dbt workflow +## Configure dbt workflow Learn how to configure the dbt workflow to ingest dbt data from your data sources. diff --git a/connectors/database/domo-database/index.mdx b/connectors/database/domo-database/index.mdx index 831d2254..b2afd5c5 100644 --- a/connectors/database/domo-database/index.mdx +++ b/connectors/database/domo-database/index.mdx @@ -29,7 +29,7 @@ For metadata ingestion, make sure to add at least `data` scopes to the clientId For questions related to scopes, click [here](https://developer.domo.com/portal/1845fc11bbe5d-api-authentication). ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/doris/index.mdx b/connectors/database/doris/index.mdx index 477575b9..8869c45d 100644 --- a/connectors/database/doris/index.mdx +++ b/connectors/database/doris/index.mdx @@ -31,7 +31,7 @@ Configure and schedule Doris metadata and profiler workflows from the Collate UI Metadata: Doris >= 1.2.0, Data Profiler: Doris >= 2.0.2 ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/dremio/index.mdx b/connectors/database/dremio/index.mdx index 39d53538..ef74c6dd 100644 --- a/connectors/database/dremio/index.mdx +++ b/connectors/database/dremio/index.mdx @@ -66,7 +66,7 @@ The connector uses Dremio's REST API and SQL interfaces for metadata extraction. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/druid/index.mdx b/connectors/database/druid/index.mdx index b60b0745..06b77464 100644 --- a/connectors/database/druid/index.mdx +++ b/connectors/database/druid/index.mdx @@ -28,7 +28,7 @@ Configure and schedule Druid metadata and profiler workflows from the Collate UI - [Troubleshooting](/connectors/database/druid/troubleshooting) ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/dynamodb/index.mdx b/connectors/database/dynamodb/index.mdx index 24e9314a..e77a8db4 100644 --- a/connectors/database/dynamodb/index.mdx +++ b/connectors/database/dynamodb/index.mdx @@ -43,7 +43,7 @@ Below defined policy grants the permissions to list all tables in DynamoDB: For more information on Dynamodb permissions visit the [AWS DynamoDB official documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/api-permissions-reference.html). ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/epic/index.mdx b/connectors/database/epic/index.mdx index 07ef531b..c5577b87 100644 --- a/connectors/database/epic/index.mdx +++ b/connectors/database/epic/index.mdx @@ -29,7 +29,7 @@ To fetch metadata from Epic FHIR into OpenMetadata you will need: 2. The FHIR version supported by your Epic server. Supported values are: `R4`, `STU3`, and `DSTU2`. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/glue/index.mdx b/connectors/database/glue/index.mdx index ea19d5db..0ec74096 100644 --- a/connectors/database/glue/index.mdx +++ b/connectors/database/glue/index.mdx @@ -27,7 +27,7 @@ Configure and schedule Glue metadata and profiler workflows from the Collate UI: User must have `glue:GetDatabases` and `glue:GetTables` permissions to ingest the basic metadata. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/greenplum/index.mdx b/connectors/database/greenplum/index.mdx index 7a2318ac..5c2a7ac6 100644 --- a/connectors/database/greenplum/index.mdx +++ b/connectors/database/greenplum/index.mdx @@ -32,7 +32,7 @@ Configure and schedule Greenplum metadata and profiler workflows from the Collat ## Requirements ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/hive/index.mdx b/connectors/database/hive/index.mdx index 679c255f..e7dfafae 100644 --- a/connectors/database/hive/index.mdx +++ b/connectors/database/hive/index.mdx @@ -35,7 +35,7 @@ To extract metadata, the user used in the connection needs to be able to perform Executing the profiler workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/how-to-guides/data-quality-observability/quality). ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/impala/index.mdx b/connectors/database/impala/index.mdx index 4de6702b..27bca716 100644 --- a/connectors/database/impala/index.mdx +++ b/connectors/database/impala/index.mdx @@ -29,7 +29,7 @@ Configure and schedule Impala metadata and profiler workflows from the Collate U - [Troubleshooting](/connectors/database/impala/troubleshooting) ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/mariadb/hybrid-runner.mdx b/connectors/database/mariadb/hybrid-runner.mdx index a216834b..09c6d96f 100644 --- a/connectors/database/mariadb/hybrid-runner.mdx +++ b/connectors/database/mariadb/hybrid-runner.mdx @@ -43,7 +43,7 @@ GRANT SELECT ON world.hello TO ''; Executing the profiler workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/how-to-guides/data-quality-observability/quality). ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Specify the User to connect to MariaDB. It should have enough privileges to read all the metadata. diff --git a/connectors/database/mariadb/index.mdx b/connectors/database/mariadb/index.mdx index 019e92fa..3d95194d 100644 --- a/connectors/database/mariadb/index.mdx +++ b/connectors/database/mariadb/index.mdx @@ -43,7 +43,7 @@ GRANT SELECT ON world.hello TO ''; Executing the profiler workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/how-to-guides/data-quality-observability/quality). ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Specify the User to connect to MariaDB. It should have enough privileges to read all the metadata. diff --git a/connectors/database/mongodb/hybrid-runner.mdx b/connectors/database/mongodb/hybrid-runner.mdx index 1fb0544d..66e15cd0 100644 --- a/connectors/database/mongodb/hybrid-runner.mdx +++ b/connectors/database/mongodb/hybrid-runner.mdx @@ -28,7 +28,7 @@ Configure and schedule MongoDB metadata workflows from the OpenMetadata UI: To fetch the metadata from MongoDB to OpenMetadata, the MongoDB user must have access to perform `find` operation on collection and `listCollection` operations on database available in MongoDB. ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Username to connect to Mongodb. This user must have access to perform `find` operation on collection and `listCollection` operations on database available in MongoDB. diff --git a/connectors/database/mongodb/index.mdx b/connectors/database/mongodb/index.mdx index 72d5389f..80a7e822 100644 --- a/connectors/database/mongodb/index.mdx +++ b/connectors/database/mongodb/index.mdx @@ -28,7 +28,7 @@ Configure and schedule MongoDB metadata workflows from the Collate UI: To fetch the metadata from MongoDB to OpenMetadata, the MongoDB user must have access to perform `find` operation on collection and `listCollection` operations on database available in MongoDB. ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Username to connect to Mongodb. This user must have access to perform `find` operation on collection and `listCollection` operations on database available in MongoDB. diff --git a/connectors/database/mssql/hybrid-runner.mdx b/connectors/database/mssql/hybrid-runner.mdx index b8341fab..7d5c440d 100644 --- a/connectors/database/mssql/hybrid-runner.mdx +++ b/connectors/database/mssql/hybrid-runner.mdx @@ -63,7 +63,7 @@ If you are using SQL server on windows, you must configure the firewall on the c For details step please refer to this [link](https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/configure-a-windows-firewall-for-database-engine-access?view=sql-server-ver15). ## Metadata Ingestion -# Connection Details +## Connection Details - **Connection Scheme**: Defines how to connect to MSSQL. We support `mssql+pytds`, `mssql+pyodbc`, and `mssql+pymssql`. (If you are using windows authentication from a linux deployment please use pymssql) diff --git a/connectors/database/mssql/index.mdx b/connectors/database/mssql/index.mdx index 04b57442..95472894 100644 --- a/connectors/database/mssql/index.mdx +++ b/connectors/database/mssql/index.mdx @@ -63,7 +63,7 @@ If you are using SQL server on windows, you must configure the firewall on the c For details step please refer to this [link](https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/configure-a-windows-firewall-for-database-engine-access?view=sql-server-ver15). ## Metadata Ingestion -# Connection Details +## Connection Details - **Connection Scheme**: Defines how to connect to MSSQL. We support `mssql+pytds`, `mssql+pyodbc`, and `mssql+pymssql`. (If you are using windows authentication from a linux deployment please use pymssql) diff --git a/connectors/database/mysql/hybrid-runner.mdx b/connectors/database/mysql/hybrid-runner.mdx index f94b209d..c3ff31bc 100644 --- a/connectors/database/mysql/hybrid-runner.mdx +++ b/connectors/database/mysql/hybrid-runner.mdx @@ -110,7 +110,7 @@ You can also check below docs about more info on logs & its rotation methods. Executing the profiler workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/how-to-guides/data-quality-observability/quality). ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Specify the User to connect to MySQL. It should have enough privileges to read all the metadata. diff --git a/connectors/database/mysql/index.mdx b/connectors/database/mysql/index.mdx index 0ee12b0d..a5ff7645 100644 --- a/connectors/database/mysql/index.mdx +++ b/connectors/database/mysql/index.mdx @@ -110,7 +110,7 @@ You can also check below docs about more info on logs & its rotation methods. Executing the profiler workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/how-to-guides/data-quality-observability/quality). ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Specify the User to connect to MySQL. It should have enough privileges to read all the metadata. diff --git a/connectors/database/oracle/hybrid-runner.mdx b/connectors/database/oracle/hybrid-runner.mdx index 0fba695f..2693bea8 100644 --- a/connectors/database/oracle/hybrid-runner.mdx +++ b/connectors/database/oracle/hybrid-runner.mdx @@ -76,7 +76,7 @@ You can find further information [here](https://docs.oracle.com/javadb/10.8.3.0/ there is no routine out of the box in Oracle to grant SELECT to a full schema. ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Specify the User to connect to Oracle. It should have enough privileges to read all the metadata. diff --git a/connectors/database/oracle/index.mdx b/connectors/database/oracle/index.mdx index 714cf5f5..308073a4 100644 --- a/connectors/database/oracle/index.mdx +++ b/connectors/database/oracle/index.mdx @@ -76,7 +76,7 @@ You can find further information [here](https://docs.oracle.com/javadb/10.8.3.0/ there is no routine out of the box in Oracle to grant SELECT to a full schema. ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Specify the User to connect to Oracle. It should have enough privileges to read all the metadata. diff --git a/connectors/database/pinotdb/hybrid-runner.mdx b/connectors/database/pinotdb/hybrid-runner.mdx index 9136ad69..fa053f60 100644 --- a/connectors/database/pinotdb/hybrid-runner.mdx +++ b/connectors/database/pinotdb/hybrid-runner.mdx @@ -34,7 +34,7 @@ To extract metadata, the user needs to have access to the Pinot broker and contr Executing the profiler workflow or data quality tests will require the user to have query execution permissions on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/how-to-guides/data-quality-observability/quality). ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Specify the User to connect to PinotDB. It should have enough privileges to read all the metadata. diff --git a/connectors/database/pinotdb/index.mdx b/connectors/database/pinotdb/index.mdx index d62a373c..f94abab2 100644 --- a/connectors/database/pinotdb/index.mdx +++ b/connectors/database/pinotdb/index.mdx @@ -33,7 +33,7 @@ To extract metadata, the user needs to have access to the Pinot broker and contr Executing the profiler workflow or data quality tests will require the user to have query execution permissions on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/how-to-guides/data-quality-observability/quality). ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Specify the User to connect to PinotDB. It should have enough privileges to read all the metadata. diff --git a/connectors/database/postgres/hybrid-runner.mdx b/connectors/database/postgres/hybrid-runner.mdx index 3afc9cd6..775f7aaa 100644 --- a/connectors/database/postgres/hybrid-runner.mdx +++ b/connectors/database/postgres/hybrid-runner.mdx @@ -98,7 +98,7 @@ By default, `pg_stat_statements` may only capture top-level procedure calls and This ensures that statements executed within procedures are recorded. ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Specify the User to connect to PostgreSQL. It should have enough privileges to read all the metadata. diff --git a/connectors/database/postgres/index.mdx b/connectors/database/postgres/index.mdx index 71abefdb..1dc15e2c 100644 --- a/connectors/database/postgres/index.mdx +++ b/connectors/database/postgres/index.mdx @@ -98,7 +98,7 @@ By default, `pg_stat_statements` may only capture top-level procedure calls and This ensures that statements executed within procedures are recorded. ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Specify the User to connect to PostgreSQL. It should have enough privileges to read all the metadata. diff --git a/connectors/database/redshift/hybrid-runner.mdx b/connectors/database/redshift/hybrid-runner.mdx index c82b458b..fd0345ce 100644 --- a/connectors/database/redshift/hybrid-runner.mdx +++ b/connectors/database/redshift/hybrid-runner.mdx @@ -58,7 +58,7 @@ For the usage and lineage workflow, the user will need `SELECT` privilege on `ST ## Metadata Ingestion It is recommmended to exclude the schema "information_schema" from the metadata ingestion as it contains system tables and views. -# Connection Details +## Connection Details diff --git a/connectors/database/redshift/index.mdx b/connectors/database/redshift/index.mdx index 7f5be556..faa118c5 100644 --- a/connectors/database/redshift/index.mdx +++ b/connectors/database/redshift/index.mdx @@ -58,7 +58,7 @@ For the usage and lineage workflow, the user will need `SELECT` privilege on `ST ## Metadata Ingestion It is recommmended to exclude the schema "information_schema" from the metadata ingestion as it contains system tables and views. -# Connection Details +## Connection Details - **Username**: Specify the User to connect to Redshift. It should have enough privileges to read all the metadata. diff --git a/connectors/database/salesforce/hybrid-runner.mdx b/connectors/database/salesforce/hybrid-runner.mdx index d08b8182..b6ca77c7 100644 --- a/connectors/database/salesforce/hybrid-runner.mdx +++ b/connectors/database/salesforce/hybrid-runner.mdx @@ -31,7 +31,7 @@ These are the permissions you will require to fetch the metadata from Salesforce - **Object Permissions**: You must have read access to the Salesforce objects that you want to ingest. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/salesforce/index.mdx b/connectors/database/salesforce/index.mdx index 592f6df2..b0671e1a 100644 --- a/connectors/database/salesforce/index.mdx +++ b/connectors/database/salesforce/index.mdx @@ -31,7 +31,7 @@ These are the permissions you will require to fetch the metadata from Salesforce - **Object Permissions**: You must have read access to the Salesforce objects that you want to ingest. ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Username to connect to the Salesforce. This user should have the access as defined in requirements. diff --git a/connectors/database/sap-erp/hybrid-runner.mdx b/connectors/database/sap-erp/hybrid-runner.mdx index 03aef08d..b9e5b51c 100644 --- a/connectors/database/sap-erp/hybrid-runner.mdx +++ b/connectors/database/sap-erp/hybrid-runner.mdx @@ -29,7 +29,7 @@ To ingest the SAP ERP metadata, CDS Views and OData services need to be setup to Follow the guide [here](/connectors/database/sap-erp/setup-sap-apis) to setup the APIs. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/sap-erp/index.mdx b/connectors/database/sap-erp/index.mdx index 22f5cf68..c895403e 100644 --- a/connectors/database/sap-erp/index.mdx +++ b/connectors/database/sap-erp/index.mdx @@ -29,7 +29,7 @@ To ingest the SAP ERP metadata, CDS Views and OData services need to be setup to Follow the guide [here](/connectors/database/sap-erp/setup-sap-apis) to setup the APIs. ## Metadata Ingestion -# Connection Details +## Connection Details - **Host and Port**: This parameter specifies the host and port of the SAP ERP instance. This should be specified as a string in the format `https://hostname.com`. diff --git a/connectors/database/sap-hana/hybrid-runner.mdx b/connectors/database/sap-hana/hybrid-runner.mdx index 9e718ba1..412347bf 100644 --- a/connectors/database/sap-hana/hybrid-runner.mdx +++ b/connectors/database/sap-hana/hybrid-runner.mdx @@ -51,7 +51,7 @@ The same applies to the `_SYS_REPO` schema, required for lineage extraction. Executing the profiler Workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. The user should also be allowed to view information in `tables` for all objects in the database. More information on the profiler workflow setup can be found [here](/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/how-to-guides/data-quality-observability/quality). ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/sap-hana/index.mdx b/connectors/database/sap-hana/index.mdx index 37106551..7b10906a 100644 --- a/connectors/database/sap-hana/index.mdx +++ b/connectors/database/sap-hana/index.mdx @@ -51,7 +51,7 @@ The same applies to the `_SYS_REPO` schema, required for lineage extraction. Executing the profiler Workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. The user should also be allowed to view information in `tables` for all objects in the database. More information on the profiler workflow setup can be found [here](/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/how-to-guides/data-quality-observability/quality). ## Metadata Ingestion -# Connection Details +## Connection Details We support two possible connection types: diff --git a/connectors/database/sas/hybrid-runner.mdx b/connectors/database/sas/hybrid-runner.mdx index 9fe45a2d..e9d14fcc 100644 --- a/connectors/database/sas/hybrid-runner.mdx +++ b/connectors/database/sas/hybrid-runner.mdx @@ -28,7 +28,7 @@ Configure and schedule SAS metadata workflow from the OpenMetadata UI: ## Metadata Ingestion Prepare the SAS Service and configure the Ingestion: -# Connection Details +## Connection Details diff --git a/connectors/database/sas/index.mdx b/connectors/database/sas/index.mdx index 83a025af..723b31a6 100644 --- a/connectors/database/sas/index.mdx +++ b/connectors/database/sas/index.mdx @@ -28,7 +28,7 @@ Configure and schedule SAS metadata workflow from the Collate UI: ## Metadata Ingestion Prepare the SAS Service and configure the Ingestion: -# Connection Details +## Connection Details - **ServerHost**: Host and port of the SAS Viya deployment. diff --git a/connectors/database/servicenow/hybrid-runner.mdx b/connectors/database/servicenow/hybrid-runner.mdx index fce6a93e..74e8fb59 100644 --- a/connectors/database/servicenow/hybrid-runner.mdx +++ b/connectors/database/servicenow/hybrid-runner.mdx @@ -30,7 +30,7 @@ To fetch metadata from ServiceNow into OpenMetadata you will need: 3. The password for the ServiceNow user. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/servicenow/index.mdx b/connectors/database/servicenow/index.mdx index 51453bca..fcf5130e 100644 --- a/connectors/database/servicenow/index.mdx +++ b/connectors/database/servicenow/index.mdx @@ -30,7 +30,7 @@ To fetch metadata from ServiceNow into OpenMetadata you will need: 3. The password for the ServiceNow user. ## Metadata Ingestion -# Connection Details +## Connection Details - **ServiceNow Instance URL**: Your ServiceNow instance URL (e.g., `https://your-instance.service-now.com`). diff --git a/connectors/database/singlestore/index.mdx b/connectors/database/singlestore/index.mdx index 7aa9faa1..a7363c72 100644 --- a/connectors/database/singlestore/index.mdx +++ b/connectors/database/singlestore/index.mdx @@ -45,7 +45,7 @@ GRANT SELECT ON world.hello TO ''; Executing the profiler workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/how-to-guides/data-quality-observability/quality). ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/snowflake/hybrid-runner.mdx b/connectors/database/snowflake/hybrid-runner.mdx index ac2f6357..c4ed1ede 100644 --- a/connectors/database/snowflake/hybrid-runner.mdx +++ b/connectors/database/snowflake/hybrid-runner.mdx @@ -74,7 +74,7 @@ You can find more information about the `account_usage` schema [here](https://do - **Ingesting Stored Procedures**: Openmetadata fetches the information by querying `snowflake.account_usage.procedures` & `snowflake.account_usage.functions`. ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Specify the User to connect to Snowflake. It should have enough privileges to read all the metadata. diff --git a/connectors/database/snowflake/index.mdx b/connectors/database/snowflake/index.mdx index 3bd3483d..ce2bbd5d 100644 --- a/connectors/database/snowflake/index.mdx +++ b/connectors/database/snowflake/index.mdx @@ -78,7 +78,7 @@ You can find more information about the `account_usage` schema [here](https://do - **Ingesting Stored Procedures**: Openmetadata fetches the information by querying `snowflake.account_usage.procedures` & `snowflake.account_usage.functions`. ## Metadata Ingestion -# Connection Details +## Connection Details - **Username**: Specify the User to connect to Snowflake. It should have enough privileges to read all the metadata. diff --git a/connectors/database/sqlite/index.mdx b/connectors/database/sqlite/index.mdx index 09c72790..60c47cd5 100644 --- a/connectors/database/sqlite/index.mdx +++ b/connectors/database/sqlite/index.mdx @@ -32,7 +32,7 @@ Configure and schedule Presto metadata and profiler workflows from the Collate U To extract metadata, the user needs to be able to perform `.tables`, `.schema`, on database you wish to extract metadata from and have `SELECT` permission on the `sqlite_temp_master`. Access to resources will be different based on the connector used. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/ssas/index.mdx b/connectors/database/ssas/index.mdx index fdeb6c80..460f4bba 100644 --- a/connectors/database/ssas/index.mdx +++ b/connectors/database/ssas/index.mdx @@ -31,7 +31,7 @@ To extract metadata from SSAS, ensure the following requirements are met: These steps are necessary to allow the connector to communicate with your SSAS instance and retrieve metadata successfully. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/teradata/index.mdx b/connectors/database/teradata/index.mdx index 7d1e8cd6..9d5e0ef3 100644 --- a/connectors/database/teradata/index.mdx +++ b/connectors/database/teradata/index.mdx @@ -34,7 +34,7 @@ Connector was tested on Teradata DBS version 17.20. Since there are no significa ## Metadata Ingestion By default, all valid users in Teradata DB has full access to metadata objects, so there are no any specific requirements to user privileges. -# Connection Details +## Connection Details diff --git a/connectors/database/trino/index.mdx b/connectors/database/trino/index.mdx index f31c2a04..6f499d40 100644 --- a/connectors/database/trino/index.mdx +++ b/connectors/database/trino/index.mdx @@ -37,7 +37,7 @@ Access to resources will be based on the user access permission to access specif Executing the profiler workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/how-to-guides/data-quality-observability/quality). ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/unity-catalog/index.mdx b/connectors/database/unity-catalog/index.mdx index 244f658d..2ecbfc9a 100644 --- a/connectors/database/unity-catalog/index.mdx +++ b/connectors/database/unity-catalog/index.mdx @@ -53,7 +53,7 @@ Adjust <user>, <catalog_name>, <schema_name>, and <table_na ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/database/vertica/index.mdx b/connectors/database/vertica/index.mdx index 15595216..f6a188c9 100644 --- a/connectors/database/vertica/index.mdx +++ b/connectors/database/vertica/index.mdx @@ -63,7 +63,7 @@ GRANT SELECT ON ALL TABLES IN SCHEMA TO openmetadata; ``` ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/messaging/kafka/index.mdx b/connectors/messaging/kafka/index.mdx index 81247a57..79930bc6 100644 --- a/connectors/messaging/kafka/index.mdx +++ b/connectors/messaging/kafka/index.mdx @@ -30,7 +30,7 @@ The ingestion of the Kafka topics' schema is done separately by configuring the - READ CLUSTER ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/messaging/kinesis/index.mdx b/connectors/messaging/kinesis/index.mdx index 3e8a7669..f5191954 100644 --- a/connectors/messaging/kinesis/index.mdx +++ b/connectors/messaging/kinesis/index.mdx @@ -47,7 +47,7 @@ The user must have the following policy set to access the metadata from Kinesis. For more information on Kinesis permissions visit the [AWS Kinesis official documentation](https://docs.aws.amazon.com/streams/latest/dev/controlling-access.html). ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/messaging/redpanda/index.mdx b/connectors/messaging/redpanda/index.mdx index 226ad909..2d99e2e1 100644 --- a/connectors/messaging/redpanda/index.mdx +++ b/connectors/messaging/redpanda/index.mdx @@ -26,7 +26,7 @@ Connecting to Redpanda does not require any previous configuration. The ingestion of the Kafka topics' schema is done separately by configuring the **Schema Registry URL**. However, only the **Bootstrap Servers** information is mandatory. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/metadata/alationsink/index.mdx b/connectors/metadata/alationsink/index.mdx index 959d98b2..e7ddfa17 100644 --- a/connectors/metadata/alationsink/index.mdx +++ b/connectors/metadata/alationsink/index.mdx @@ -41,7 +41,7 @@ Following entities are supported and will be mapped to the from OpenMetadata to ## Metadata Ingestion Then, prepare the Alation Sink Service and configure the Ingestion: -# Connection Details +## Connection Details diff --git a/connectors/metadata/amundsen/index.mdx b/connectors/metadata/amundsen/index.mdx index ca21162c..7c182f6f 100644 --- a/connectors/metadata/amundsen/index.mdx +++ b/connectors/metadata/amundsen/index.mdx @@ -18,7 +18,7 @@ availableFeatures={["Metadata"]} unavailableFeatures={[]} /> ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/metadata/atlas/index.mdx b/connectors/metadata/atlas/index.mdx index 5605c9a1..3b21db28 100644 --- a/connectors/metadata/atlas/index.mdx +++ b/connectors/metadata/atlas/index.mdx @@ -29,7 +29,7 @@ For example, to create a Hive Service you can follow these steps: ## 2. Atlas Metadata Ingestion Then, prepare the Atlas Service and configure the Ingestion: -# Connection Details +## Connection Details diff --git a/connectors/ml-model/mlflow/index.mdx b/connectors/ml-model/mlflow/index.mdx index 88c4cee2..aa1c922e 100644 --- a/connectors/ml-model/mlflow/index.mdx +++ b/connectors/ml-model/mlflow/index.mdx @@ -27,7 +27,7 @@ To extract metadata, OpenMetadata needs two elements: - **Registry URI**: Address of local or remote model registry server. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/ml-model/sagemaker/index.mdx b/connectors/ml-model/sagemaker/index.mdx index ea71eef9..17a77f9e 100644 --- a/connectors/ml-model/sagemaker/index.mdx +++ b/connectors/ml-model/sagemaker/index.mdx @@ -48,7 +48,7 @@ SageMaker also supports metadata ingestion of SageMaker Unified Studio models. T For more information on Sagemaker permissions visit the [AWS Sagemaker official documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/api-permissions-reference.html). ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/airbyte/index.mdx b/connectors/pipeline/airbyte/index.mdx index 42264fa3..373ee2a9 100644 --- a/connectors/pipeline/airbyte/index.mdx +++ b/connectors/pipeline/airbyte/index.mdx @@ -22,7 +22,7 @@ Configure and schedule Airbyte metadata and profiler workflows from the Collate - [Troubleshooting](/connectors/pipeline/airbyte/troubleshooting) ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/airflow/gcp-composer.mdx b/connectors/pipeline/airflow/gcp-composer.mdx index 1422950a..aa6ffbaf 100644 --- a/connectors/pipeline/airflow/gcp-composer.mdx +++ b/connectors/pipeline/airflow/gcp-composer.mdx @@ -249,7 +249,7 @@ KubernetesPodOperator( You can find more information about the `KubernetesPodOperator` and how to tune its configurations [here](https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator). -# OpenMetadata Server Config +## OpenMetadata Server Config The easiest approach here is to generate a bot with a **JWT** token directly from the Collate UI. You can then use the following workflow config: diff --git a/connectors/pipeline/airflow/index.mdx b/connectors/pipeline/airflow/index.mdx index 4452f235..28d769a3 100644 --- a/connectors/pipeline/airflow/index.mdx +++ b/connectors/pipeline/airflow/index.mdx @@ -35,7 +35,7 @@ You can check the version list [here](https://airflow.apache.org/docs/apache-air ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/dagster/index.mdx b/connectors/pipeline/dagster/index.mdx index a07f6500..f2dd9e80 100644 --- a/connectors/pipeline/dagster/index.mdx +++ b/connectors/pipeline/dagster/index.mdx @@ -30,7 +30,7 @@ OpenMetadata is integrated with dagster up to version [1.0.13](https://docs.dags The ingestion framework uses [dagster graphql python client](https://docs.dagster.io/_apidocs/libraries/dagster-graphql#dagster_graphql.DagsterGraphQLClient) to connect to the dagster instance and perform the API calls ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/databricks-pipeline/index.mdx b/connectors/pipeline/databricks-pipeline/index.mdx index 180e1f52..517d808d 100644 --- a/connectors/pipeline/databricks-pipeline/index.mdx +++ b/connectors/pipeline/databricks-pipeline/index.mdx @@ -22,7 +22,7 @@ Configure and schedule Databricks Pipeline metadata workflows from the Collate U - [Troubleshooting](/connectors/pipeline/databricks-pipeline/troubleshooting) ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/datafactory/index.mdx b/connectors/pipeline/datafactory/index.mdx index c009f114..b4e8e6e1 100644 --- a/connectors/pipeline/datafactory/index.mdx +++ b/connectors/pipeline/datafactory/index.mdx @@ -33,7 +33,7 @@ You can find further information on the Azure Data Factory connector in the [doc Ensure that the service principal or managed identity you’re using has the necessary permissions in the Data Factory resource (Reader, Contributor or Data Factory Contributor role at minimum). ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/dbtcloud/index.mdx b/connectors/pipeline/dbtcloud/index.mdx index 4b2021d2..cd095ba5 100644 --- a/connectors/pipeline/dbtcloud/index.mdx +++ b/connectors/pipeline/dbtcloud/index.mdx @@ -37,7 +37,7 @@ To know more about permissions required refer [here](https://docs.getdbt.com/doc - Your projects must be on dbt version 1.0 or later. Refer to [Upgrade dbt version in Cloud](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-version-in-cloud) to upgrade. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/domo-pipeline/index.mdx b/connectors/pipeline/domo-pipeline/index.mdx index cf23181d..1ca1499a 100644 --- a/connectors/pipeline/domo-pipeline/index.mdx +++ b/connectors/pipeline/domo-pipeline/index.mdx @@ -26,7 +26,7 @@ For metadata ingestion, make sure to add at least `data` scopes to the clientId For questions related to scopes, click [here](https://developer.domo.com/portal/1845fc11bbe5d-api-authentication). ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/fivetran/index.mdx b/connectors/pipeline/fivetran/index.mdx index 0e0708bd..98cbcdfb 100644 --- a/connectors/pipeline/fivetran/index.mdx +++ b/connectors/pipeline/fivetran/index.mdx @@ -25,7 +25,7 @@ Configure and schedule Fivetran metadata and profiler workflows from the Collate To access Fivetran APIs, a Fivetran account on a Standard, Enterprise, or Business Critical plan is required. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/flink/index.mdx b/connectors/pipeline/flink/index.mdx index fcf0fa34..574fc4c2 100644 --- a/connectors/pipeline/flink/index.mdx +++ b/connectors/pipeline/flink/index.mdx @@ -30,7 +30,7 @@ OpenMetadata is integrated with flink up to version [1.19.0](https://nightlies.a The ingestion framework uses flink REST APIs to connect to the instance and perform the API calls ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/glue-pipeline/index.mdx b/connectors/pipeline/glue-pipeline/index.mdx index fed92b0e..e5de21f1 100644 --- a/connectors/pipeline/glue-pipeline/index.mdx +++ b/connectors/pipeline/glue-pipeline/index.mdx @@ -30,7 +30,7 @@ The user must have the following permissions for the ingestion to run successful - `glue:GetJobRuns` ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/kafkaconnect/index.mdx b/connectors/pipeline/kafkaconnect/index.mdx index a4b1cc64..5e817a26 100644 --- a/connectors/pipeline/kafkaconnect/index.mdx +++ b/connectors/pipeline/kafkaconnect/index.mdx @@ -30,7 +30,7 @@ OpenMetadata is integrated with kafkaconnect up to version [3.6.1](https://docs. The ingestion framework uses [kafkaconnect python client](https://libraries.io/pypi/kafka-connect-py) to connect to the kafkaconnect instance and perform the API calls ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/matillion/index.mdx b/connectors/pipeline/matillion/index.mdx index 1ca84b62..fe8cf0b0 100644 --- a/connectors/pipeline/matillion/index.mdx +++ b/connectors/pipeline/matillion/index.mdx @@ -33,7 +33,7 @@ To extract metadata from Matillion, you need to create a user with the following OpenMetadata is integrated with matillion up to version [1.75.0](https://docs.matillion.io/getting-started). ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/nifi/index.mdx b/connectors/pipeline/nifi/index.mdx index e1a9cbca..cac29296 100644 --- a/connectors/pipeline/nifi/index.mdx +++ b/connectors/pipeline/nifi/index.mdx @@ -29,7 +29,7 @@ OpenMetadata supports 2 types of connection for the NiFi connector: The user should be able to send request to the NiFi API and access the `Resources` endpoint. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/snowplow/index.mdx b/connectors/pipeline/snowplow/index.mdx index 2c8f6d22..f0b3c650 100644 --- a/connectors/pipeline/snowplow/index.mdx +++ b/connectors/pipeline/snowplow/index.mdx @@ -37,7 +37,7 @@ For self-hosted Snowplow Community deployments, you need: - The file system path to your configuration files ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/spline/index.mdx b/connectors/pipeline/spline/index.mdx index 4230420d..c05a44d6 100644 --- a/connectors/pipeline/spline/index.mdx +++ b/connectors/pipeline/spline/index.mdx @@ -29,7 +29,7 @@ Currently, we do not support data source of type AWS S3 or any other cloud stora You can refer [this](https://github.com/AbsaOSS/spline-getting-started/tree/main/spline-on-databricks) documentation on how to configure Databricks with Spline. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/ssis/index.mdx b/connectors/pipeline/ssis/index.mdx index c2dfd5f6..76e05006 100644 --- a/connectors/pipeline/ssis/index.mdx +++ b/connectors/pipeline/ssis/index.mdx @@ -28,7 +28,7 @@ To extract SSIS metadata, we need the batabase connection details where the meta - To retrieve lineage data, the user must be granted [Component-level permissions](https://docs.matillion.com/metl/docs/2932106/#component). ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/stitch/index.mdx b/connectors/pipeline/stitch/index.mdx index e0d98709..03033032 100644 --- a/connectors/pipeline/stitch/index.mdx +++ b/connectors/pipeline/stitch/index.mdx @@ -27,7 +27,7 @@ To extract metadata from Stitch, User first need to crate API crednetials: - `Token`: Token to access Stitch metadata. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/pipeline/wherescape/index.mdx b/connectors/pipeline/wherescape/index.mdx index b8d1873c..f08dbfc2 100644 --- a/connectors/pipeline/wherescape/index.mdx +++ b/connectors/pipeline/wherescape/index.mdx @@ -28,7 +28,7 @@ To extract Wherescape metadata, we need the batabase connection details where th - To retrieve lineage data, the user must be granted [Component-level permissions](https://docs.matillion.com/metl/docs/2932106/#component). ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/search/elasticsearch/index.mdx b/connectors/search/elasticsearch/index.mdx index 50158389..52db5f25 100644 --- a/connectors/search/elasticsearch/index.mdx +++ b/connectors/search/elasticsearch/index.mdx @@ -26,7 +26,7 @@ We support Elasticsearch 7.0 and above. We extract Elasticsearch's metadata by using its [API](https://www.elastic.co/guide/en/elasticsearch/reference/current/rest-apis.html). To run this ingestion, you just need a user with permissions to the ElasticSearch instance. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/connectors/search/opensearch/index.mdx b/connectors/search/opensearch/index.mdx index 8310e646..11d4f6c2 100644 --- a/connectors/search/opensearch/index.mdx +++ b/connectors/search/opensearch/index.mdx @@ -25,7 +25,7 @@ Configure and schedule OpenSearch metadata workflow from the Collate UI: We extract OpenSearch's metadata by using its [API](https://opensearch.org/docs/latest/api-reference/). To run this ingestion, you just need a user with permissions to the OpenSearch instance. ## Metadata Ingestion -# Connection Details +## Connection Details diff --git a/deployment/backup-restore-metadata.mdx b/deployment/backup-restore-metadata.mdx index 81f69a02..385e8fe7 100644 --- a/deployment/backup-restore-metadata.mdx +++ b/deployment/backup-restore-metadata.mdx @@ -53,13 +53,13 @@ You can check these two examples on how to: - Use pipes to stream the result directly to S3 (or AWS blob storage) ([link](https://devcoops.com/pg_dump-to-s3-directly/?utm_content=cmp-true)). - Dump to a file and copy to storage ([link](https://gist.github.com/bbcoimbra/0914c7e0f96e8ad53dfad79c64863c87)). -# Example with Docker +## Example with Docker Start a local instance of OpenMetadata using the `docker-compose` file provided in the repository. Then, we can use the following commands to backup the metadata: -## MySQL +### MySQL -### 1. Backup and Restore +#### 1. Backup and Restore ```shell BACKUP_FILE="backup_$(date +%Y%m%d%H%M).sql" @@ -75,16 +75,16 @@ docker compose exec mysql mysql -u root -ppassword -e "flush privileges;" docker compose exec -T ingestion mysql -u openmetadata_user -popenmetadata_password -h mysql -P 3306 restore < $BACKUP_FILE ``` -### 2. Restart the docker deployment with the restored database +#### 2. Restart the docker deployment with the restored database ```shell export OM_DATABASE=restore docker compose -f $DOCKER_COMPOSE_FILE up -d ``` -## PostgreSQL +### PostgreSQL -### 1. Backup and Restore +#### 1. Backup and Restore ```shell BACKUP_FILE="backup_$(date +%Y%m%d%H%M).sql" @@ -98,7 +98,7 @@ docker compose exec -e PGPASSWORD=openmetadata_password postgresql psql -U postg docker compose exec -e PGPASSWORD=openmetadata_password -T ingestion psql -U openmetadata_user -h postgresql -d restore < $BACKUP_FILE ``` -### 2. Restart the docker deployment with the restored database +#### 2. Restart the docker deployment with the restored database ```shell export OM_DATABASE=restore diff --git a/deployment/bare-metal/index.mdx b/deployment/bare-metal/index.mdx index a13b5cab..aa88e1dc 100644 --- a/deployment/bare-metal/index.mdx +++ b/deployment/bare-metal/index.mdx @@ -78,16 +78,16 @@ OpenMetadata versions have specific Airflow compatibility requirements to ensure These settings apply as well when using managed instances, such as RDS or AWS OpenSearch. -# Procedure +## Procedure -## 1. Download the distribution +### 1. Download the distribution Visit the [releases page](https://github.com/open-metadata/OpenMetadata/releases/latest) and download the latest binary release. Release binaries follow the naming convention of `openmetadata-x.y.z.tar.gz`. Where `x`, `y`, and `z` represent the major, minor, and patch release numbers. -## 2. Untar the release download +### 2. Untar the release download Once the tar file has downloaded, run the following command, updated if necessary for the version of OpenMetadata that you downloaded. @@ -95,7 +95,7 @@ Once the tar file has downloaded, run the following command, updated if necessar tar -zxvf openmetadata-*.tar.gz ``` -## 3. Navigate to the directory created +### 3. Navigate to the directory created ```commandline cd openmetadata-* @@ -105,7 +105,7 @@ Review and update the `openmetadata.yaml` configurations to match your environme as the connection to the MySQL database or ElasticSearch. You can find more information about these configurations [here](/deployment/configuration). -## 4. Prepare the OpenMetadata Database and Indexes +### 4. Prepare the OpenMetadata Database and Indexes The command below will generate all the necessary tables and indexes in ElasticSearch. @@ -119,7 +119,7 @@ Note that if there's any data in that database, this command will drop it! ./bootstrap/openmetadata-ops.sh drop-create ``` -## 5. Start OpenMetadata +### 5. Start OpenMetadata ```commandline ./bin/openmetadata.sh start diff --git a/deployment/index.mdx b/deployment/index.mdx index 669946c0..a511faa5 100644 --- a/deployment/index.mdx +++ b/deployment/index.mdx @@ -20,7 +20,7 @@ Are you exploring or doing a PoC? It won't get easier than following our Quickst -# Architecture and Deployment +## Architecture and Deployment We support different kinds of deployment: @@ -36,7 +36,7 @@ We support different kinds of deployment: -# Security +## Security Find further information on how to secure your OpenMetadata deployment: @@ -46,7 +46,7 @@ Find further information on how to secure your OpenMetadata deployment: -# Production-Ready Requirements +## Production-Ready Requirements Find further information on how to secure your OpenMetadata deployment: @@ -56,7 +56,7 @@ Find further information on how to secure your OpenMetadata deployment: -# Upgrade +## Upgrade Learn how to move from version to version: diff --git a/deployment/ingestion/openmetadata.mdx b/deployment/ingestion/openmetadata.mdx index 3b5576d8..17fcfbf5 100644 --- a/deployment/ingestion/openmetadata.mdx +++ b/deployment/ingestion/openmetadata.mdx @@ -333,11 +333,11 @@ If you want to learn how to set up Airflow using SSL, you can learn more here: -# Troubleshooting +## Troubleshooting -## Ingestion Pipeline deployment issues +### Ingestion Pipeline deployment issues -### Airflow APIs Not Found +#### Airflow APIs Not Found Validate the installation, making sure that from the OpenMetadata server you can reach the Airflow host, and the call to `/health` gives us the proper response: @@ -349,7 +349,7 @@ $ curl -XGET ${PIPELINE_SERVICE_CLIENT_ENDPOINT}/api/v1/openmetadata/health Also, make sure that the version of your OpenMetadata server matches the `openmetadata-ingestion` client version installed in Airflow. -### GetServiceException: Could not get service from type XYZ +#### GetServiceException: Could not get service from type XYZ In this case, the OpenMetadata client running in the Airflow host had issues getting the service you are trying to deploy from the API. Note that once pipelines are deployed, the auth happens via the `ingestion-bot`. Here there are @@ -370,7 +370,7 @@ a couple of points to validate: {"code":403,"message":"Principal: CatalogPrincipal{name='ingestion-bot'} operations [ViewAll] not allowed"} ``` -### AirflowException: Dag 'XYZ' could not be found +#### AirflowException: Dag 'XYZ' could not be found If you're seeing a similar error to @@ -389,7 +389,7 @@ in your distributed environment. We have specific instructions on how to set up the shared volumes in Kubernetes depending on your cloud deployment [here](/deployment/kubernetes). -### ClientInitializationError +#### ClientInitializationError The main root cause here is a version mismatch between the server and the client. Make sure that the `openmetadata-ingestion` python package you installed on the Airflow host has the same version as the OpenMetadata server. For example, to set up @@ -397,7 +397,7 @@ OpenMetadata server 0.13.2 you will need to install `openmetadata-ingestion~=0.1 the version as in `x.y.z`. Any differences after the PATCH versioning are not taken into account, as they are usually small bugfixes on existing functionalities. -### 401 Unauthorized +#### 401 Unauthorized If you get this response during a `Test Connection` or `Deploy`: @@ -420,7 +420,7 @@ pipelineServiceClientConfiguration: You should validate if the content of the environment variables `AIRFLOW_USERNAME` and `AIRFLOW_PASSWORD` allow you to authenticate to the instance. -### CentOS / Debian - The name 'template_blueprint' is already registered +#### CentOS / Debian - The name 'template_blueprint' is already registered If you are using a CentOS / Debian system to install the `openmetadata-managed-apis` you might encounter the following issue when starting Airflow: diff --git a/deployment/security/basic-auth/index.mdx b/deployment/security/basic-auth/index.mdx index 5dead741..3406e950 100644 --- a/deployment/security/basic-auth/index.mdx +++ b/deployment/security/basic-auth/index.mdx @@ -38,13 +38,13 @@ Security requirements for your **production** environment: -# Setting up Basic Auth Manually +## Setting up Basic Auth Manually Below are the required steps to set up the Basic Login: -## Set up Configurations in openmetadata.yaml +### Set up Configurations in openmetadata.yaml -### Authentication Configuration +#### Authentication Configuration The following configuration controls the auth mechanism for OpenMetadata. Update the mentioned fields as required. @@ -63,7 +63,7 @@ For the Basic auth we need to set: - `authority`: {your_domain} - `enableSelfSignup`: This flag indicates if users can come and signup by themselves on the OM -### Authorizer Configuration +#### Authorizer Configuration This configuration controls the authorizer for OpenMetadata: @@ -92,7 +92,7 @@ After logging into the OpenMetadata UI, admin users can change their default pas -## Metadata Ingestion +### Metadata Ingestion For ingesting metadata when Basic Auth is enabled, it is mandatory to configure the `ingestion-bot` account with the JWT configuration. To know how to enable it, you can follow the documentation of [Enable JWT Tokens](/deployment/security/enable-jwt-tokens). diff --git a/deployment/security/enable-ssl/airflow-ssl.mdx b/deployment/security/enable-ssl/airflow-ssl.mdx index 147834f4..435e8e00 100644 --- a/deployment/security/enable-ssl/airflow-ssl.mdx +++ b/deployment/security/enable-ssl/airflow-ssl.mdx @@ -37,7 +37,7 @@ Note that the `PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH` should be the path to the in your Airflow deployment. -# Enable SSL in Airflow +## Enable SSL in Airflow @@ -53,7 +53,7 @@ Airflow has two [configurations](https://airflow.apache.org/docs/apache-airflow/ Those are files that will need to be local to the Airflow deployment. -## Generate Certs +### Generate Certs We can generate these files following this [SO](https://stackoverflow.com/questions/47883769/how-to-enable-ssl-on-apache-airflow) thread: @@ -83,7 +83,7 @@ It is important that the `Common Name` is the host name that will be hosting Air This command will generate the pair `airflow.key` and `airflow.crt`. -## Include Certificates +### Include Certificates Once the files are generated we need to add them to the Airflow deployment. For example, if using the `openmetadata-ingestion` image, you can update it to add the following lines: @@ -98,7 +98,7 @@ ENV AIRFLOW__WEBSERVER__WEB_SERVER_SSL_KEY=/opt/airflow/airflow.key If you now start Airflow with these changes, it will be running at `https://localhost:8080`. -## Update the OpenMetadata configuration +### Update the OpenMetadata configuration Since Airflow will be using SSL, we need to update the OpenMetadata Server configuration to use the certificates when preparing the connection to the Airflow Webserver. diff --git a/deployment/upgrade/bare-metal.mdx b/deployment/upgrade/bare-metal.mdx index af1c9018..815b94a9 100644 --- a/deployment/upgrade/bare-metal.mdx +++ b/deployment/upgrade/bare-metal.mdx @@ -20,9 +20,9 @@ This guide assumes that you have an OpenMetadata deployment that you installed a -# Upgrade process +## Upgrade process -## Step 1: Download the binaries for the release you want to install +### Step 1: Download the binaries for the release you want to install OpenMetadata release binaries are maintained as GitHub releases. @@ -34,7 +34,7 @@ To download a specific release binary: - Download the release binaries. The release binaries will be in a compressed tar file named using the following convention, `openmetadata-x.y.z.tar.gz` Where `x`, `y`, `z` are the major, minor, and patch release numbers, respectively. -## Step 2: Extract the release binaries from the download file +### Step 2: Extract the release binaries from the download file Using the command-line tool or application of your choice, extract the release binaries. @@ -46,7 +46,7 @@ tar xfz openmetadata-*.tar.gz This will create a directory with the same name as the download file minus the `.tar` and `.gz` extensions. -## Step 3: Navigate into the directory created by extracting the release binaries +### Step 3: Navigate into the directory created by extracting the release binaries Change into the new directory by issuing a command similar to the following. @@ -61,7 +61,7 @@ command. cd openmetadata-1.1.0 ``` -## Step 4: Stop the OpenMetadata server +### Step 4: Stop the OpenMetadata server OpenMetadata ships with a few control scripts. One is `openmetadata.sh`. This script enables you to start, stop, and perform other deployment operations on the OpenMetadata server. @@ -75,7 +75,7 @@ directory of your current installation by running the following command: ./bin/openmetadata.sh stop ``` -## Step 5: Migrate the database schemas and ElasticSearch indexes +### Step 5: Migrate the database schemas and ElasticSearch indexes The `bootstrap/openmetadata-ops.sh` script enables you to perform a number of operations on the OpenMetadata database (in MySQL) and index (in Elasticsearch). @@ -85,7 +85,7 @@ MySQL) and index (in Elasticsearch). ``` -## Step 6: Restart the OpenMetadata server +### Step 6: Restart the OpenMetadata server Once you've dropped and recreated your data in the new version, restart the OpenMetadata server using the new release binaries. You may restart the server by running the following command. diff --git a/deployment/upgrade/docker.mdx b/deployment/upgrade/docker.mdx index 1f45adc0..d1bf03a7 100644 --- a/deployment/upgrade/docker.mdx +++ b/deployment/upgrade/docker.mdx @@ -24,9 +24,9 @@ Below we have highlighted the steps needed to upgrade to the latest version with -# Upgrade Process +## Upgrade Process -## Replace the docker compose file +### Replace the docker compose file - Stop the running compose deployment with below command ``` @@ -42,7 +42,7 @@ docker compose -f docker-compose.yml up -d -# Troubleshooting +## Troubleshooting #### Permission Denied when running ```metadata openmetadata-imports-migration``` If you have a `Permission Denied` error thrown when running ```metadata openmetadata-imports-migration --change-config-file-path``` you might need to change the permission on the `/opt/airflow/dags` folder. SSH into the ingestion container and check the permission on the folder running the below commands diff --git a/deployment/upgrade/index.mdx b/deployment/upgrade/index.mdx index 5c04ff52..a7e428fe 100644 --- a/deployment/upgrade/index.mdx +++ b/deployment/upgrade/index.mdx @@ -15,7 +15,7 @@ In this guide, you will find all the necessary information to safely upgrade you -# Upgrade your installation +## Upgrade your installation Once your metadata is safe, follow the required upgrade instructions based on your environment: diff --git a/deployment/upgrade/kubernetes.mdx b/deployment/upgrade/kubernetes.mdx index b0fcc234..cb6d3869 100644 --- a/deployment/upgrade/kubernetes.mdx +++ b/deployment/upgrade/kubernetes.mdx @@ -22,15 +22,15 @@ We also assume that your helm chart release names are `openmetadata` and `openme -# Upgrade Process +## Upgrade Process -## Step 1: Get an overview of what has changed in Helm Values +### Step 1: Get an overview of what has changed in Helm Values You can get changes from artifact hub of [openmetadata helm chart](https://artifacthub.io/packages/helm/open-metadata/openmetadata) release. Click on Default Values >> Compare to Version. Helm Chart Release Comparison -## Step 2: Upgrade Helm Repository with a new release +### Step 2: Upgrade Helm Repository with a new release Update Helm Chart Locally for OpenMetadata with the below command: @@ -60,7 +60,7 @@ open-metadata/openmetadata-dependencies 1.2.8 1.2.5 Helm Dep ... ``` -## Step 3: Upgrade OpenMetadata Dependencies +### Step 3: Upgrade OpenMetadata Dependencies You can run the below command to upgrade the dependencies with the new chart @@ -80,7 +80,7 @@ If you are running into any issues, double-check what are the default values of -## Step 4: Upgrade OpenMetadata +### Step 4: Upgrade OpenMetadata Finally, we upgrade OpenMetadata with the below command: @@ -102,7 +102,7 @@ You can learn more about how the migration process works [here](/deployment/upgr -# Troubleshooting +## Troubleshooting ### Helm Upgrade fails with additional property airflow not allowed diff --git a/developers/webhooks.mdx b/developers/webhooks.mdx index a05c8cfd..282dd36a 100644 --- a/developers/webhooks.mdx +++ b/developers/webhooks.mdx @@ -18,7 +18,7 @@ Please use the same webhook config that you had saved from previous version to c -# Webhooks +## Webhooks A webhook is a way for an app to provide other applications with real-time information. A webhook delivers data to other applications as it happens, meaning you get data immediately. @@ -26,13 +26,13 @@ OpenMetadata provides out-of-the-box support for webhooks. OpenMetadata also allows the user to customise the webhook with a wide range of filters to listen to only selected type of events. -## OpenMetadata supports 4 webhook types: +### OpenMetadata supports 4 webhook types: 1. **Generic** 2. **Slack** 3. **Microsoft Teams** 4. **Google Chat** -## How to Set up Generic Type Webhook: +### How to Set up Generic Type Webhook: 1. **Name**: Add the name of the webhook 2. **Description**: Describe the webhook. 3. **Destination**: Enter the **consumer or client URL** where the event data should be delivered. You can also configure the **Connection Timeout (seconds)** and **Read Timeout (seconds)** to manage request-response limits for the webhook call. @@ -43,7 +43,7 @@ OpenMetadata also allows the user to customise the webhook with a wide range of Generic Webhook -## How to Set up Slack Type Webhook: +### How to Set up Slack Type Webhook: 1. **Name**: Add the name of the webhook 2. **Description**: Describe the webhook. 3. **Destination**: Enter the Slack endpoint URL where the webhook data should be sent. This is the target endpoint that will receive the event payload. Ensure the Slack webhook URL is correctly configured to enable successful message delivery. @@ -55,7 +55,7 @@ OpenMetadata also allows the user to customise the webhook with a wide range of Slack Webhook -## How to Set up Microsoft Teams Type Webhook: +### How to Set up Microsoft Teams Type Webhook: 1. **Name**: Add the name of the webhook 2. **Description**: Describe the webhook. 3. **Destination**: Enter the MS Teams endpoint URL. You can find this by going to the Teams channel where you want the posts to appear, clicking the three dots `...`, and clicking "Connectors". Then add the "Incoming Webhook" connector. Copy this connector's URL and supply it here to OpenMetadata. It may be in the form of `https://your-domain.webhook.office.com/webhookb2/...@.../IncomingWebhook/.../...`. For more on MS Teams webhooks, see [Create an Incoming Webhook](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook). @@ -66,7 +66,7 @@ OpenMetadata also allows the user to customise the webhook with a wide range of MS Team Webhook -## How to Set up Google Chat Type Webhook: +### How to Set up Google Chat Type Webhook: 1. **Name**: Add the name of the webhook 2. **Description**: Describe the webhook. 3. **Destination**: Enter the GChat endpoint URL. For more on creating GChat webhooks, see [Create a Webhook](https://developers.google.com/chat/how-tos/webhooks#create_a_webhook). diff --git a/how-to-guides/data-contracts/spec.mdx b/how-to-guides/data-contracts/spec.mdx index 934dffb5..970e2b1f 100644 --- a/how-to-guides/data-contracts/spec.mdx +++ b/how-to-guides/data-contracts/spec.mdx @@ -11,7 +11,7 @@ Data contracts formalize an agreement between data producers and consumers about Collate, as a metadata management platform, integrates this concept by introducing a Data Contract entity defined via JSON Schema. This allows Collate admins and [data product](https://docs.open-metadata.org/latest/how-to-guides/data-governance/domains-&-data-products#data-products) owners to attach a contract to tables in Collate, codifying expectations in a structured format. The contract can then be enforced or validated using Collate’s existing metadata and data quality frameworks. The goal is to have contextually rich, high-quality, well-governed data that is trustworthy. Data contracts achieve this by making data expectations explicit and automating their enforcement. -# Data Contract Entity Schema Design +## Data Contract Entity Schema Design The JSON Schema definition for Data Contract entities in Collate defines the contract’s structure and allowed fields. The contract covers four main categories of expectations: @@ -27,30 +27,30 @@ We also include an SLA section for service-level agreements and an Ownership fie The JSON Schema for the Data Contract entity can be found [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/data/dataContract.json) -# Data Contract Sections -## Schema +## Data Contract Sections +### Schema This is where the expected structural schema of the data asset is defined. It includes a list of fields (for a table, these are columns) each with name and data type. This captures the contractual schema that producers and consumers agreed on, which usually is a subset of the available fields on an asset. -## Semantics +### Semantics Business meaning and documentation requirements are defined in a contract's Semantics section. For example, one can enforce that a data asset must have a description, owner or domain. These rules complement the formal tests in the quality section, acting as documentation of business expectations. This section ensures the contract isn’t just about technical schema, but also carries business context. -## Security +### Security Data security and access expectations are defined in this section. This can reference an access policy ID or name that should govern this data, or a required classification label. In practice, this means the contract might require the data asset to be tagged as `PII` or `Confidential` if appropriate, and that only certain roles can access it (through an associated policy). -## Quality (Assertions) +### Quality (Assertions) Data quality tests and assertions required by the contract are here. Built on top of the native Data Quality features in Collate, this section allows defining specific tests that must pass for the data to be considered compliant with the contract. Tests can be at the column level (e.g., a column must be non-null) or table level (e.g., row count must be above a threshold), and can be managed from the Data Contract UI itself. -## SLA +### SLA Service-Level Agreements related to the data’s timeliness and lifecycle are captured in this section. This includes: @@ -74,11 +74,11 @@ A status field indicates whether the contract is active, draft, or currently vio Data Contracts have approval workflows when changes are made, similar to Glossaries. This ensures that any modifications to the contract (like adding new quality tests or changing schema expectations) go through a review and approval process. -# Applying Contracts to Tables +## Applying Contracts to Tables Below is an example of a data contract for a warehouse.sales.orders table. -## Data Contract YAML Example +### Data Contract YAML Example This is an example YAML of a Data Contract applied to a table in Collate. Note that while Collate brings full UI support for creating and managing Data Contracts, you can still use the API to manage them programmatically. diff --git a/how-to-guides/data-discovery/collate-extension/index.mdx b/how-to-guides/data-discovery/collate-extension/index.mdx index 3110bd8f..4ee42919 100644 --- a/how-to-guides/data-discovery/collate-extension/index.mdx +++ b/how-to-guides/data-discovery/collate-extension/index.mdx @@ -16,7 +16,7 @@ sidebarTitle: Overview allowFullScreen> -# Access Metadata Without Leaving Your Workspace +## Access Metadata Without Leaving Your Workspace The Collate Chrome extension provides a seamless way to interact with metadata directly within your workspace. It allows users to view activity feeds, resolve assigned tasks, and look up data assets without switching between tools or interfaces. This integration enhances productivity by providing instant access to metadata and task management features right from your browser. diff --git a/how-to-guides/data-discovery/details.mdx b/how-to-guides/data-discovery/details.mdx index 2be03d50..dd387fca 100644 --- a/how-to-guides/data-discovery/details.mdx +++ b/how-to-guides/data-discovery/details.mdx @@ -11,7 +11,7 @@ The data asset details page displays the **Source, Owner (Team/User), Tier, Type Overview of Data Assets -# Version History and Other Details +## Version History and Other Details On the top right of the data asset details page, we can view details on: - **Tasks:** The circular icon displays the number of open tasks. @@ -24,7 +24,7 @@ On the top right of the data asset details page, we can view details on: Version History and Other Details -# Data Asset Tabs +## Data Asset Tabs There are separate tabs each for Schema, Activity Feeds & Tasks, Sample Data, Queries, Data Observability, Lineage, Custom Properties, Config, Details, Features, Children, and Executions based on the type of data asset selected. Let's take a look at each of the tabs. | **TABS** | **Table** | **Topic** | **Dashboard** | **Pipeline** | **ML Model** | **Container** | @@ -42,31 +42,31 @@ There are separate tabs each for Schema, Activity Feeds & Tasks, Sample Data, Qu | **Features** | | | | | | | | **Children** | | | | | | | -## Schema Tab +### Schema Tab The Schema Data tab is displayed only for Tables, Topics, and Containers. Schema will display the columns, type of column, and description, alongwith the tags, and glossary terms associated with each column. The table also displays details on the **Frequently Joined Tables, Tags, and Glossary Terms** associated with it. Schema Tab -## Activity Feeds & Tasks Tab +### Activity Feeds & Tasks Tab The Activity Feeds & Task tab is displayed for all types of data assets. It displays all the tasks and mentions for a particular data asset. Activity Feeds & Tasks Tab -## Sample Data Tab +### Sample Data Tab During metadata ingestion, you can opt to bring in sample data. If sample data is enabled, the same is displayed here. The Sample Data tab is displayed only for Tables and Topics. Sample Data Tab -## Queries Tab +### Queries Tab The Queries tab is displayed only for Tables. It displays the SQL queries run against a particular table. It provides the details on when the query was run and the amount of time taken. It also displays if the query was used by other tables. You can also add new queries. Queries Tab -## Data Observability Tab +### Data Observability Tab The Data Observability tab is displayed only for Tables. It has three sub-tabs for **Table Profile, Column Profile, and Data Quality**. The Data Observability brings in details like number of rows and columns for the table profile alongwith the details of the data volume, table updates, and volume changes. For the column profile, it provides details about the type of each column, the value count, null value %, distinct value %, unique %, etc. Data quality tests can be run on this sample data, and tests can be added at both the table and column levels. @@ -74,7 +74,7 @@ The Data Observability tab is displayed only for Tables. It has three sub-tabs f Column Profile of a Table -## Lineage Tab +### Lineage Tab The lineage tab is displayed for all types of data assets. The lineage view displays comprehensive lineage to capture the relation between the data assets. Collate UI displays end-to-end lineage traceability for the table and column levels. It displays both the upstream and downstream for each node. @@ -84,17 +84,17 @@ Users can configure the number of upstreams, downstreams, and nodes per layer by Data Asset Preview in Lineage Tab -## Custom Properties Tab +### Custom Properties Tab Collate uses a schema-first approach. We also support custom properties for all types of data assets. Organizations can extend the attributes as required to capture custom metadata. The Custom Properties tab shows up for all types of data assets. User can add or edit the custom property values for the data assets from this tab. Learn [How to Create a Custom Property for a Data Asset](/how-to-guides/guide-for-data-users/custom) Enter the Value for a Custom Property -## Config Tab +### Config Tab The Config tab is displayed only for Topics. -## Details Tab +### Details Tab The Details tab is displayed only for Dashboards and ML Models. In case of Dashboards, the Details tab displays the chart name, type of chart, and description of the chart. It also displays the associated tags for each chart. Dashboards: Details Tab @@ -102,19 +102,19 @@ The Details tab is displayed only for Dashboards and ML Models. In case of Dashb In case of ML Models, it displays the Hyper Parameters and Model Store details. ML Models: Details Tab -## Executions Tab +### Executions Tab The Executions tab is displayed only for Pipelines. It displays the Date, Time, and Status of the pipelines. You can get a quick glance of the status in terms of Success, Failure, Pending, and Aborted. The status can be viewed as a Chronological list or as a tree. You can filter by status as well as by date. Pipelines: Executions Tab -## Features Tab +### Features Tab The Features tab is displayed only for ML Models. It displays a Description of the ML Model, and the features that have been used. Each feature will have further details on the Type of feature, Algorithm, Description, Sources, and the associated Glossary Terms and Tags. ML Models: Features Tab -## Children Tab +### Children Tab The Children tab is displayed only for Containers. diff --git a/how-to-guides/data-governance/workflows/default-workflows/dashboard-certification.mdx b/how-to-guides/data-governance/workflows/default-workflows/dashboard-certification.mdx index cab0a6f3..c50c6c2d 100644 --- a/how-to-guides/data-governance/workflows/default-workflows/dashboard-certification.mdx +++ b/how-to-guides/data-governance/workflows/default-workflows/dashboard-certification.mdx @@ -72,7 +72,7 @@ Certification workflows can be triggered: dashboard-certification-trigger -# Benefits +## Benefits - **Automated Trust Indicators**: Helps users easily identify dashboards that are well-maintained. - **Metadata Quality Assurance**: Encourages teams to provide complete documentation and ownership. diff --git a/how-to-guides/data-governance/workflows/default-workflows/glossary-approval.mdx b/how-to-guides/data-governance/workflows/default-workflows/glossary-approval.mdx index 8e8ef490..dcf62bfc 100644 --- a/how-to-guides/data-governance/workflows/default-workflows/glossary-approval.mdx +++ b/how-to-guides/data-governance/workflows/default-workflows/glossary-approval.mdx @@ -42,7 +42,7 @@ A business glossary is critical for ensuring consistent terminology across an or - If the Glossary Term is edited by a reviewer, then the Glossary Term is Auto Approved without the requirement of any approval. This can be modified to suit the needs of the organization. -# Approval and Rejection Threshold for User Tasks +## Approval and Rejection Threshold for User Tasks - Flexibility to set the Desired Approval and Rejection Threshold for User Approval Tasks. diff --git a/how-to-guides/data-insights/service-insights.mdx b/how-to-guides/data-insights/service-insights.mdx index 5606e3ab..5270e14f 100644 --- a/how-to-guides/data-insights/service-insights.mdx +++ b/how-to-guides/data-insights/service-insights.mdx @@ -61,7 +61,7 @@ If no data is displayed, verify that both the metadata ingestion pipeline and th Platform Insights -# Generated Data with Collate AI (Collate Only) +## Generated Data with Collate AI (Collate Only) This table displays a breakdown of metadata populated by the Collate AI agent versus metadata populated manually. diff --git a/how-to-guides/guide-for-data-users/data-asset-tabs.mdx b/how-to-guides/guide-for-data-users/data-asset-tabs.mdx index 57a277b7..985c6d23 100644 --- a/how-to-guides/guide-for-data-users/data-asset-tabs.mdx +++ b/how-to-guides/guide-for-data-users/data-asset-tabs.mdx @@ -5,7 +5,7 @@ slug: /how-to-guides/guide-for-data-users/data-asset-tabs sidebarTitle: Overview of Data Assets --- -# Overview of Data Assets +## Overview of Data Assets Collate displays a single-pane view for each of the data assets. In the detailed view of a data asset, the **Source, Owner (Team/User), Tier, Type, Usage, Description** are displayed on the top panel. Further, there are separate tabs each for Schema, Activity Feeds & Tasks, Sample Data, Queries, Profiler & Data Quality, Lineage, Custom Properties, Config, Details, Features, Children, and Executions based on the type of data asset selected. @@ -108,7 +108,7 @@ The Features tab is displayed only for ML Models. It displays a Description of t The Children tab is displayed only for Containers. -# Version History and Other Details +## Version History and Other Details On the top right of the data asset details page, we can view details on: - **Tasks:** The circular icon displays the number of open tasks. diff --git a/how-to-guides/guide-for-data-users/ingestion/best-practices.mdx b/how-to-guides/guide-for-data-users/ingestion/best-practices.mdx index 8d1f88b1..d46cf597 100644 --- a/how-to-guides/guide-for-data-users/ingestion/best-practices.mdx +++ b/how-to-guides/guide-for-data-users/ingestion/best-practices.mdx @@ -52,7 +52,7 @@ We will use the generic terms from Airflow, as the most common used tool, but th look for the past week, as we will be re-analysing data that won’t change. -# OpenMetadata Ingestion Troubleshooting +## OpenMetadata Ingestion Troubleshooting Here we will discuss different errors that you might encounter when running a workflow: diff --git a/how-to-guides/sql-studio/user-authentication.mdx b/how-to-guides/sql-studio/user-authentication.mdx index 08168c33..665ac9fe 100644 --- a/how-to-guides/sql-studio/user-authentication.mdx +++ b/how-to-guides/sql-studio/user-authentication.mdx @@ -52,7 +52,7 @@ The authentication method available to you is determined by your administrator. --- -# Collate SSO Authentication +## Collate SSO Authentication @@ -139,7 +139,7 @@ Collate SSO leverages your existing Google OIDC SSO configuration in Collate to --- -# OAuth Authentication +## OAuth Authentication @@ -147,7 +147,7 @@ Collate SSO leverages your existing Google OIDC SSO configuration in Collate to -## How OAuth Works +### How OAuth Works OAuth requires you to authenticate directly with the database service provider (Google for BigQuery, Snowflake, Azure AD for Trino). You'll be redirected to the provider's authentication page, grant permissions, and then SQL Studio securely stores your OAuth tokens for query execution. @@ -156,15 +156,15 @@ OAuth requires you to authenticate directly with the database service provider ( - Direct authentication with database provider - Works with any OAuth provider configured by your administrator -## Connecting with OAuth +### Connecting with OAuth -### Step 1: Access SQL Studio +#### Step 1: Access SQL Studio 1. Click **SQL Studio** from the main navigation menu 2. You'll see a landing page with available services 3. Find the service you want to connect to -### Step 2: Select Service +#### Step 2: Select Service 1. Click on the service card (e.g., BigQuery, Snowflake, Trino) 2. A connection dialog appears @@ -177,7 +177,7 @@ OAuth requires you to authenticate directly with the database service provider ( - **Snowflake**: Snowflake authentication page (e.g., `myorg.snowflakecomputing.com/oauth/authorize`) - **Trino**: Azure AD or configured OAuth provider -### Step 3: Authenticate with Provider +#### Step 3: Authenticate with Provider 1. Sign in with your credentials: - **BigQuery**: Google email and password @@ -186,7 +186,7 @@ OAuth requires you to authenticate directly with the database service provider ( 2. Complete any multi-factor authentication if required 3. You may be asked to select an account or role (follow the prompts) -### Step 4: Grant Permissions +#### Step 4: Grant Permissions 1. Review the permissions requested: - **BigQuery**: "Allow access to BigQuery API" @@ -194,7 +194,7 @@ OAuth requires you to authenticate directly with the database service provider ( - **Trino**: "Allow access to Trino/Starburst cluster" 2. Click **Allow**, **Accept**, or **Authorize** (wording varies by provider) -### Step 5: Connection Established +#### Step 5: Connection Established 1. After successful authentication, access token exchange happens automatically 2. You're redirected back to SQL Studio @@ -204,7 +204,7 @@ OAuth requires you to authenticate directly with the database service provider ( 6. After a few seconds, status changes to **Connected** 🟢 7. A new query tab opens automatically -### Step 6: Start Querying +#### Step 6: Start Querying 1. The SQL Editor is now active 2. You can write and execute queries @@ -216,9 +216,9 @@ OAuth requires you to authenticate directly with the database service provider ( - Database connection parameters pulled from service configuration - Connection tested automatically -## Troubleshooting External OAuth +### Troubleshooting External OAuth -### "Redirect URI Mismatch" Error +#### "Redirect URI Mismatch" Error **Problem**: Error message about redirect URI mismatch during OAuth flow @@ -226,7 +226,7 @@ OAuth requires you to authenticate directly with the database service provider ( - Contact your administrator - OAuth app configuration needs correction - Verify you're using the correct Collate URL (e.g., `https://collate.example.com`, not `http://localhost`) -### Authentication Successful But Connection Fails +#### Authentication Successful But Connection Fails **Problem**: OAuth completes but test connection fails @@ -235,7 +235,7 @@ OAuth requires you to authenticate directly with the database service provider ( - Check network connectivity to the database service - Contact your administrator to review service configuration -### Redirect Issues +#### Redirect Issues **Problem**: Not redirected back to SQL Studio after authentication @@ -246,7 +246,7 @@ OAuth requires you to authenticate directly with the database service provider ( --- -# Basic Authentication +## Basic Authentication @@ -254,7 +254,7 @@ OAuth requires you to authenticate directly with the database service provider ( -## How Basic Authentication Works +### How Basic Authentication Works Basic Authentication requires you to manually enter credentials such as Username + Password, Service Account JSON, or Key Pair. Credentials are encrypted and stored securely. @@ -265,9 +265,9 @@ Basic Authentication requires you to manually enter credentials such as Username --- -## BigQuery Basic Authentication +### BigQuery Basic Authentication -### Credentials Required +#### Credentials Required - **GCP Service Account JSON** file, containing: - Private Key ID @@ -275,7 +275,7 @@ Basic Authentication requires you to manually enter credentials such as Username - Client Email - Client ID -### Where to Get Credentials +#### Where to Get Credentials 1. Go to [Google Cloud Console](https://console.cloud.google.com/) 2. Navigate to **IAM & Admin** → **Service Accounts** @@ -286,14 +286,14 @@ Basic Authentication requires you to manually enter credentials such as Username 5. Go to **Keys** tab → **Add Key** → **Create new key** → **JSON** 6. Download the JSON file and store securely (e.g., 1Password) -### Connecting with BigQuery Basic Auth +#### Connecting with BigQuery Basic Auth -#### Step 1: Select Service +##### Step 1: Select Service 1. In SQL Studio, select the BigQuery service from the dropdown 2. Click **Configure Connection** -#### Step 2: Enter Credentials +##### Step 2: Enter Credentials You have two options: @@ -312,13 +312,13 @@ You have two options: - **Client Email**: Copy from `client_email` field - **Client ID**: Copy from `client_id` field -#### Step 3: (Optional) Configure Overrides +##### Step 3: (Optional) Configure Overrides If your administrator enabled user-configurable fields: - **Dataset**: Enter default BigQuery dataset (e.g., `analytics`, `marketing`) -#### Step 4: Test Connection +##### Step 4: Test Connection 1. Click **Test Connection** 2. SQL Studio validates your credentials by: @@ -327,7 +327,7 @@ If your administrator enabled user-configurable fields: 3. Connection status changes to **Pending** 🟡 4. Wait for test to complete -#### Step 5: Save Configuration +##### Step 5: Save Configuration 1. If test succeeds, click **Save** 2. Your credentials are encrypted and stored diff --git a/quick-start/local-docker-deployment.mdx b/quick-start/local-docker-deployment.mdx index 9fd6d7d0..16edca9f 100644 --- a/quick-start/local-docker-deployment.mdx +++ b/quick-start/local-docker-deployment.mdx @@ -14,14 +14,14 @@ content below 👇 -# Requirements (OSX, Linux and Windows) +## Requirements (OSX, Linux and Windows) Please ensure your host system meets the requirements listed below. Then continue to the Procedure for installing OpenMetadata. -## OSX and Linux +### OSX and Linux -### Docker (version 20.10.0 or greater) +#### Docker (version 20.10.0 or greater) [Docker](https://docs.docker.com/get-started/overview/) is an open-source platform for developing, shipping, and running applications. It enables you to separate your applications from your infrastructure, so you can deliver software quickly using OS-level virtualization. It helps deliver software in packages called Containers. @@ -40,7 +40,7 @@ You must allocate at least `6 GiB` of memory and `4 vCPUs` to Docker in order to -### Docker Compose (version v2.1.1 or greater) +#### Docker Compose (version v2.1.1 or greater) The Docker `compose` package enables you to define and run multi-container Docker applications. The compose command integrates compose functions into the Docker platform, making them available from the Docker command-line interface ( @@ -62,7 +62,7 @@ Upon running this command you should see output similar to the following. Docker Compose version v2.1.1 ``` -### Install Docker Compose Version 2.0.0 on Linux +#### Install Docker Compose Version 2.0.0 on Linux Follow the instructions [here](https://docs.docker.com/compose/cli-command/#install-on-linux) to install docker compose version 2.0.0 @@ -88,9 +88,9 @@ Follow the instructions [here](https://docs.docker.com/compose/cli-command/#inst > Docker Compose version v2.2.3 ``` -## Windows +### Windows -### WSL2, Ubuntu 20.04, and Docker for Windows +#### WSL2, Ubuntu 20.04, and Docker for Windows - Install [WSL2](https://ubuntu.com/wsl) - Install [Ubuntu 20.04](https://www.microsoft.com/en-us/p/ubuntu-2004-lts/9n6svws3rx71) @@ -100,9 +100,9 @@ Follow the instructions [here](https://docs.docker.com/compose/cli-command/#inst - Command: `apt install python3-pip python3-venv` (Ensure that you have the privilege to install packages, if not, please use Super User.) -## Procedure +### Procedure -### 1. Create a directory for OpenMetadata +#### 1. Create a directory for OpenMetadata Create a new directory for OpenMetadata and navigate into that directory. @@ -110,7 +110,7 @@ Create a new directory for OpenMetadata and navigate into that directory. mkdir openmetadata-docker && cd openmetadata-docker ``` -### 2. Download Docker Compose File from GitHub Releases +#### 2. Download Docker Compose File from GitHub Releases Download the docker-compose.yml file from the release page [here](https://github.com/open-metadata/OpenMetadata/releases/latest). @@ -202,22 +202,22 @@ You can access Airflow at [http://localhost:8080](http://localhost:8080). Use th - Username: `admin` - Password: `admin` -### Customizing Airflow Admin Credentials: +#### Customizing Airflow Admin Credentials: When using Docker Compose, you can change the default Airflow admin credentials by setting the following environment variables: - Username: `AIRFLOW_ADMIN_USER` - Password: `AIRFLOW_ADMIN_PASSWORD` -## Airflow DAGs Showcased in Deployment +### Airflow DAGs Showcased in Deployment You can explore the examples of Airflow DAGs used with OpenMetadata. Refer [here](https://github.com/open-metadata/OpenMetadata/tree/main/ingestion/examples/airflow/dags) for more information. DAG_Examples -## Go on a tour and start discovering the power of metadata & collaboration +### Go on a tour and start discovering the power of metadata & collaboration tour -## Start and Stop +### Start and Stop From the same directory mentioned in [step 1](#1.-create-a-directory-for-openmetadata), use the following commands to start and stop the Docker Compose services. diff --git a/snippets/connectors/troubleshooting.mdx b/snippets/connectors/troubleshooting.mdx index 019e998c..96071841 100644 --- a/snippets/connectors/troubleshooting.mdx +++ b/snippets/connectors/troubleshooting.mdx @@ -1,6 +1,6 @@ -# Troubleshooting +## Troubleshooting -## Workflow Deployment Error +### Workflow Deployment Error If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be diff --git a/snippets/deployment/upgrade/post-upgrade-steps.mdx b/snippets/deployment/upgrade/post-upgrade-steps.mdx index 6c52313c..39691470 100644 --- a/snippets/deployment/upgrade/post-upgrade-steps.mdx +++ b/snippets/deployment/upgrade/post-upgrade-steps.mdx @@ -1,7 +1,7 @@ import Reindex from '/snippets/deployment/reindex.mdx' import Redeploy from '/snippets/deployment/redeploy.mdx' -# Post-Upgrade Steps +## Post-Upgrade Steps ### Reindex @@ -132,13 +132,13 @@ kubectl logs job/ If you are seeing broken dags select all the pipelines from all the services and re deploy the pipelines. -# Openmetadata-ops Script +## Openmetadata-ops Script -## Overview +### Overview The `openmetadata-ops` script is designed to manage and migrate databases and search indexes, reindex existing data into Elastic Search or OpenSearch, and redeploy service pipelines. -## Usage +### Usage ``` bash sh openmetadata-ops.sh [-dhV] [COMMAND] diff --git a/snippets/deployment/upgrade/upgrade-prerequisites.mdx b/snippets/deployment/upgrade/upgrade-prerequisites.mdx index 258aa4f9..a1b7c92d 100644 --- a/snippets/deployment/upgrade/upgrade-prerequisites.mdx +++ b/snippets/deployment/upgrade/upgrade-prerequisites.mdx @@ -1,8 +1,8 @@ -# Prerequisites +## Prerequisites Everytime that you plan on upgrading OpenMetadata to a newer version, make sure to go over all these steps: -## Backup your Metadata +### Backup your Metadata Before upgrading your OpenMetadata version we strongly recommend backing up the metadata. @@ -42,7 +42,7 @@ You can refer to the following guide to get more details about the backup and re -## Understanding the "Running" State in OpenMetadata +### Understanding the "Running" State in OpenMetadata In OpenMetadata, the **"Running"** state indicates that the Collate server has received a response from Airflow confirming that a workflow is in progress. However, if Airflow unexpectedly stops or crashes before it can send a failure status update through the **Failure Callback**, OpenMetadata remains unaware of the workflow’s actual state. As a result, the workflow may appear to be stuck in **"Running"** even though it is no longer executing. @@ -56,7 +56,7 @@ To resolve this issue: - Manually update the pipeline status if necessary. - Check Airflow logs to verify if the DAG execution was interrupted. -### Update `sort_buffer_size` (MySQL) or `work_mem` (Postgres) +#### Update `sort_buffer_size` (MySQL) or `work_mem` (Postgres) Before running the migrations, it is important to update these parameters to ensure there are no runtime errors. A safe value would be setting them to 20MB. @@ -94,16 +94,16 @@ during the migration after bumping this value, you can increase them further. After the migration is finished, you can revert this changes. -# Backward Incompatible Changes +## Backward Incompatible Changes -## 1.9.0 +### 1.9.0 -### Strong validation of test case parameters +#### Strong validation of test case parameters `parameterValues` name of a testCase will be strongly validated against the name of the `parameterDefinition` in the testDefinition. If both parameter names do not match an error will be thrown on testCase creation -### Multi-domain Support +#### Multi-domain Support **All entities now support multi-domains**. Their `domain` field is now renamed to `domains` and modelled as a list of domains instead of a single domain.