From 618af3d4c2fde5f177f3faf325dec5241e53c2b6 Mon Sep 17 00:00:00 2001 From: galilev Date: Tue, 30 Jul 2024 16:02:40 +0300 Subject: [PATCH 1/5] adding an example for adding signal table and a new table to the pipeline (Oracle) --- .../adding-tables-to-existing-pipeline.md | 73 +++++++++++++++++-- 1 file changed, 67 insertions(+), 6 deletions(-) diff --git a/content/rdi/installation/adding-tables-to-existing-pipeline.md b/content/rdi/installation/adding-tables-to-existing-pipeline.md index 6c742cd3fde..8ea3ae8a58c 100644 --- a/content/rdi/installation/adding-tables-to-existing-pipeline.md +++ b/content/rdi/installation/adding-tables-to-existing-pipeline.md @@ -5,7 +5,7 @@ description: Set up and use Debezium to add additional tables to an existing pip weight: 80 alwaysopen: false categories: ["redis-di"] -aliases: +aliases: --- If you want to add a new table to a pipeline that is already in streaming (CDC) mode, you can do so without resetting Debezium Server and executing a new full snapshot. In Debezium, this is called incremental snapshot and it is performed using a table on the source database as the interface with the Debezium connector. @@ -114,14 +114,75 @@ The data-collections array lists tables by their fully-qualified names, using th #### Signaling Table Columns -| Column | Description | -| ------ | --------------------------------------------------------------------------------- | -| id | An arbitrary string that is assigned as the identifier for the signal request. | -| type | The type of signal to send. | -| data | An array of table names to include in the snapshot. | +| Column | Description | +| ------ | ------------------------------------------------------------------------------ | +| id | An arbitrary string that is assigned as the identifier for the signal request. | +| type | The type of signal to send. | +| data | An array of table names to include in the snapshot. | ## SQL Server: `sp_cdc_enable_table` Stored Procedure Arguments - `@source_name` - Specifies the name of the table that you want to capture. - `@role_name` - Specifies a role MyRole to which you can add users to whom you want to grant SELECT permission on the captured columns of the source table. Users in the sysadmin or db_owner role also have access to the specified change tables. Set the value of @role_name to NULL, to allow only members in the sysadmin or db_owner to have full access to captured information. - `@filegroup_name` - Specifies the filegroup where SQL Server places the change table for the captured table. The named filegroup must be already exist. It is best not to locate change tables in the same filegroup that you use for source tables. + +## Example for Adding a Signaling Table for Oracle Database + +1. Creating a signaling table `DEBEZIUM_SIGNAL`: + + ```sql + CREATE TABLE DEBEZIUM_SIGNAL + ( + id VARCHAR(42) PRIMARY KEY, + type VARCHAR(32) NOT NULL, + data VARCHAR(2048) NULL + ); + ``` + +2. Adding the property `debezium.source.signal.data.collection` to the `application.properties` file: + + ```properties + debezium.source.signal.data.collection=ORCLPDB1.C##DBZUSER.DEBEZIUM_SIGNAL + ``` + + > Note: The property `debezium.source.signal.data.collection` should be set to the fully qualified name of the table. In `Oracle`, the fully qualified name includes the schema name `C##DBZUSER` and the database name `ORCLPDB1`. + +3. Enable supplemental logging for the `DEBEZIUM_SIGNAL` table: + + ``` + ALTER table c##dbzuser.DEBEZIUM_SIGNAL add SUPPLEMENTAL LOG DATA(ALL) COLUMNS; + ``` + + > Note: If the supplemental logging is enabled for the entire database you can skip this step. + +4. Restart the Debezium Server + +## Example for Adding the EMP table to the pipeline + +1. Add the `EMP` table to the `debezium.source.table.include.list` property in the `application.properties` file: + + ```properties + debezium.source.table.include.list=C##DBZUSER.EMP + ``` + +2. Enable supplemental logging for the `EMP`` table: + + To enable supplemental logging for all the table columns: + + ```sql + ALTER Table c##dbzuser.EMP add SUPPLEMENTAL LOG DATA(ALL) + COLUMNS + ``` + +3. Restart the `Debezium Server` + +4. To trigger the `incremental snapshot` for the `EMP table, run: + + ```sql + INSERT INTO c##dbzuser.DEBEZIUM_SIGNAL ds (id, type, data) + VALUES ('1', 'execute-snapshot', '{"data-collections":["ORCLPDB1.C##DBZUSER.EMP"],"type":"incremental"}'); + ``` + + > Note: The column `id` is a unique string in the DEBEZIUM_SIGNAL table + +5. The `EMP` table will be added to the pipeline, and irs keys will be stored in the RDI bdb with no need to run `redis-di reset` From 58c319c341713d607d31b709e9b6882e080a7878 Mon Sep 17 00:00:00 2001 From: galilev Date: Tue, 30 Jul 2024 16:08:39 +0300 Subject: [PATCH 2/5] add an example for adding a signal table and a new table to the pipeline(Oracle) --- .../adding-tables-to-existing-pipeline.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/content/rdi/installation/adding-tables-to-existing-pipeline.md b/content/rdi/installation/adding-tables-to-existing-pipeline.md index 8ea3ae8a58c..5c710d58cfc 100644 --- a/content/rdi/installation/adding-tables-to-existing-pipeline.md +++ b/content/rdi/installation/adding-tables-to-existing-pipeline.md @@ -128,7 +128,7 @@ The data-collections array lists tables by their fully-qualified names, using th ## Example for Adding a Signaling Table for Oracle Database -1. Creating a signaling table `DEBEZIUM_SIGNAL`: +1. Create a signaling table `DEBEZIUM_SIGNAL`: ```sql CREATE TABLE DEBEZIUM_SIGNAL @@ -139,7 +139,7 @@ The data-collections array lists tables by their fully-qualified names, using th ); ``` -2. Adding the property `debezium.source.signal.data.collection` to the `application.properties` file: +2. Add the property `debezium.source.signal.data.collection` to the `application.properties` file: ```properties debezium.source.signal.data.collection=ORCLPDB1.C##DBZUSER.DEBEZIUM_SIGNAL @@ -149,13 +149,13 @@ The data-collections array lists tables by their fully-qualified names, using th 3. Enable supplemental logging for the `DEBEZIUM_SIGNAL` table: - ``` + ```sql ALTER table c##dbzuser.DEBEZIUM_SIGNAL add SUPPLEMENTAL LOG DATA(ALL) COLUMNS; ``` > Note: If the supplemental logging is enabled for the entire database you can skip this step. -4. Restart the Debezium Server +4. Restart the Debezium Server. ## Example for Adding the EMP table to the pipeline @@ -170,11 +170,11 @@ The data-collections array lists tables by their fully-qualified names, using th To enable supplemental logging for all the table columns: ```sql - ALTER Table c##dbzuser.EMP add SUPPLEMENTAL LOG DATA(ALL) + ALTER TABLE c##dbzuser.EMP ADD SUPPLEMENTAL LOG DATA(ALL) COLUMNS ``` -3. Restart the `Debezium Server` +3. Restart the `Debezium Server`. 4. To trigger the `incremental snapshot` for the `EMP table, run: @@ -185,4 +185,4 @@ The data-collections array lists tables by their fully-qualified names, using th > Note: The column `id` is a unique string in the DEBEZIUM_SIGNAL table -5. The `EMP` table will be added to the pipeline, and irs keys will be stored in the RDI bdb with no need to run `redis-di reset` +5. The `EMP` table will be added to the pipeline, and irs keys will be stored in the RDI bdb with no need to run `redis-di reset`. From 7faf4caad52a740f804dab00433e61a1b79c20ee Mon Sep 17 00:00:00 2001 From: galilev Date: Tue, 30 Jul 2024 16:17:04 +0300 Subject: [PATCH 3/5] -Adding to the doc an example for adding signaling table and a new table to pipeline --- .../adding-tables-to-existing-pipeline.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/content/rdi/installation/adding-tables-to-existing-pipeline.md b/content/rdi/installation/adding-tables-to-existing-pipeline.md index 5c710d58cfc..49fe1bc8569 100644 --- a/content/rdi/installation/adding-tables-to-existing-pipeline.md +++ b/content/rdi/installation/adding-tables-to-existing-pipeline.md @@ -159,30 +159,29 @@ The data-collections array lists tables by their fully-qualified names, using th ## Example for Adding the EMP table to the pipeline -1. Add the `EMP` table to the `debezium.source.table.include.list` property in the `application.properties` file: +1. Add the `CUSTOMERS` table to the `debezium.source.table.include.list` property in the `application.properties` file: ```properties - debezium.source.table.include.list=C##DBZUSER.EMP + debezium.source.table.include.list=C##DBZUSER.PRODUCTS,C##DBZUSER.ORDERS,C##DBZUSER.CUSTOMERS ``` -2. Enable supplemental logging for the `EMP`` table: +2. Enable supplemental logging for the `CUSTOMERS`` table: To enable supplemental logging for all the table columns: ```sql - ALTER TABLE c##dbzuser.EMP ADD SUPPLEMENTAL LOG DATA(ALL) - COLUMNS + ALTER TABLE C##DBZUSER.CUSTOMERS ADD SUPPLEMENTAL LOG DATA(ALL) COLUMNS ``` 3. Restart the `Debezium Server`. -4. To trigger the `incremental snapshot` for the `EMP table, run: +4. To trigger the `incremental snapshot` for the `CUSTOMERS` table, run: ```sql INSERT INTO c##dbzuser.DEBEZIUM_SIGNAL ds (id, type, data) - VALUES ('1', 'execute-snapshot', '{"data-collections":["ORCLPDB1.C##DBZUSER.EMP"],"type":"incremental"}'); + VALUES ('1', 'execute-snapshot', '{"data-collections":["ORCLPDB1.C##DBZUSER.CUSTOMERS"],"type":"incremental"}'); ``` > Note: The column `id` is a unique string in the DEBEZIUM_SIGNAL table -5. The `EMP` table will be added to the pipeline, and irs keys will be stored in the RDI bdb with no need to run `redis-di reset`. +5. The `CUSTOMERS` table will be added to the pipeline, and irs keys will be stored in the RDI bdb with no need to run `redis-di reset`. From 6c1ab3037c4bac535a276324ad71a3037a8c3e5c Mon Sep 17 00:00:00 2001 From: galilev Date: Tue, 30 Jul 2024 16:18:14 +0300 Subject: [PATCH 4/5] Adding an examole to the doc for adding signaling table and a new table to the pipeline --- content/rdi/installation/adding-tables-to-existing-pipeline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/rdi/installation/adding-tables-to-existing-pipeline.md b/content/rdi/installation/adding-tables-to-existing-pipeline.md index 49fe1bc8569..843887408ac 100644 --- a/content/rdi/installation/adding-tables-to-existing-pipeline.md +++ b/content/rdi/installation/adding-tables-to-existing-pipeline.md @@ -157,7 +157,7 @@ The data-collections array lists tables by their fully-qualified names, using th 4. Restart the Debezium Server. -## Example for Adding the EMP table to the pipeline +## Example for Adding the `CUSTOMERS` table to the pipeline 1. Add the `CUSTOMERS` table to the `debezium.source.table.include.list` property in the `application.properties` file: From 9f5d1bb8f15df44c0ac47902dea3a75e77cc110e Mon Sep 17 00:00:00 2001 From: galilev Date: Tue, 30 Jul 2024 16:19:54 +0300 Subject: [PATCH 5/5] syntax err --- content/rdi/installation/adding-tables-to-existing-pipeline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/rdi/installation/adding-tables-to-existing-pipeline.md b/content/rdi/installation/adding-tables-to-existing-pipeline.md index 843887408ac..b9bf5afbfde 100644 --- a/content/rdi/installation/adding-tables-to-existing-pipeline.md +++ b/content/rdi/installation/adding-tables-to-existing-pipeline.md @@ -184,4 +184,4 @@ The data-collections array lists tables by their fully-qualified names, using th > Note: The column `id` is a unique string in the DEBEZIUM_SIGNAL table -5. The `CUSTOMERS` table will be added to the pipeline, and irs keys will be stored in the RDI bdb with no need to run `redis-di reset`. +5. The `CUSTOMERS` table will be added to the pipeline, and its keys will be stored in the RDI bdb with no need to run `redis-di reset`.