Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added Azure Synapse Analytics to supported products #1

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Added Azure Synapse Analytics to supported products #1

wants to merge 2 commits into from

Conversation

t3t5u
Copy link

@t3t5u t3t5u commented May 20, 2024 •
edited
Loading

config.yml 
$ cat config.yml
in:
  type: sqlserver
  product: azure_synapse_analytics
  host: test-synapse-workspace0.sql.azuresynapse.net
  user: test_user@test-synapse-workspace0
  password: ***
  database: test_dedicated_sql_pool
  query: SELECT * FROM [test_schema].[test_table] ORDER BY [id]
  column_options:
    test_timestamp: {type: string, timestamp_format: '%Y-%m-%d %H:%M:%S.%6N %z'}
out:
  type: command
  command: 'cat -'
  formatter:
    type: jsonl
run 
$ embulk run config.yml
2024-05-22 00:46:59.716 +0000: Embulk v0.9.26
2024-05-22 00:47:00.310 +0000 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
2024-05-22 00:47:01.825 +0000 [INFO] (main): Gem's home and path are set by default: "/home/ubuntu/.embulk/lib/gems"
2024-05-22 00:47:02.301 +0000 [INFO] (main): Started Embulk v0.9.26
2024-05-22 00:47:02.367 +0000 [INFO] (0001:transaction): Loaded plugin embulk-input-sqlserver (0.13.2)
2024-05-22 00:47:02.392 +0000 [INFO] (0001:transaction): Loaded plugin embulk-output-command (0.1.4)
2024-05-22 00:47:02.419 +0000 [WARN] (0001:transaction): "UTC" is recognized as "Z" to be compatible with the legacy style.
2024-05-22 00:47:02.430 +0000 [INFO] (0001:transaction): Connecting to jdbc:sqlserver://test-synapse-workspace0.sql.azuresynapse.net:1433 options {hostNameInCertificate=*.sql.azuresynapse.net, socketTimeout=1800000, user=test_user@test-synapse-workspace0, encrypt=true, databaseName=test_dedicated_sql_pool, trustServerCertificate=false, applicationName=embulk-input-sqlserver, password=***, loginTimeout=300}
2024-05-22 00:47:02.788 +0000 [INFO] (0001:transaction): Using JDBC Driver 7.2.2.0
2024-05-22 00:47:03.013 +0000 [INFO] (0001:transaction): Using local thread executor with max_threads=16 / output tasks 8 = input tasks 1 * 8
2024-05-22 00:47:03.040 +0000 [INFO] (0001:transaction): Loaded plugin embulk-formatter-jsonl (0.1.4)
2024-05-22 00:47:03.062 +0000 [INFO] (0001:transaction): {done:  0 / 1, running: 0}
2024-05-22 00:47:03.069 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:47:03.080 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:47:03.084 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:47:03.088 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:47:03.092 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:47:03.095 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:47:03.101 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:47:03.106 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:47:03.123 +0000 [WARN] (0015:task-0000): Z is deprecated as a military time zone name. Use UTC instead.
2024-05-22 00:47:03.123 +0000 [WARN] (0015:task-0000): "Z" is recognized as "Z" to be compatible with the legacy style.
2024-05-22 00:47:03.149 +0000 [INFO] (0015:task-0000): Connecting to jdbc:sqlserver://test-synapse-workspace0.sql.azuresynapse.net:1433 options {hostNameInCertificate=*.sql.azuresynapse.net, socketTimeout=1800000, user=test_user@test-synapse-workspace0, encrypt=true, databaseName=test_dedicated_sql_pool, trustServerCertificate=false, applicationName=embulk-input-sqlserver, password=***, loginTimeout=300}
2024-05-22 00:47:03.285 +0000 [INFO] (0015:task-0000): SQL: SELECT * FROM [test_schema].[test_table] ORDER BY [id]
2024-05-22 00:47:03.464 +0000 [INFO] (0015:task-0000): > 0.18 seconds
{"id":1,"test_boolean":true,"test_long":123,"test_double":1.23,"test_string":"あいうえお","test_timestamp":"1999-12-31 23:59:59.000000 +0000","test_json":"{\"キー\":\"値\"}","test_json_text":"{\"キー\":\"値\"}","test_json_string":"{\"キー\":\"値\"}","test_json_nstring":"{\"キー\":\"値\"}"}
{"id":2,"test_boolean":false,"test_long":456,"test_double":4.56,"test_string":"かきくけこ","test_timestamp":"2000-01-01 00:00:00.000000 +0000","test_json":"[{\"キー1\":\"値1\"},{\"キー2\":\"値2\"}]","test_json_text":"[{\"キー1\":\"値1\"},{\"キー2\":\"値2\"}]","test_json_string":"[{\"キー1\":\"値1\"},{\"キー2\":\"値2\"}]","test_json_nstring":"[{\"キー1\":\"値1\"},{\"キー2\":\"値2\"}]"}
{"id":3,"test_boolean":null,"test_long":null,"test_double":null,"test_string":null,"test_timestamp":null,"test_json":null,"test_json_text":null,"test_json_string":null,"test_json_nstring":null}
{"id":4,"test_boolean":null,"test_long":null,"test_double":null,"test_string":null,"test_timestamp":null,"test_json":"12345","test_json_text":"12345","test_json_string":"12345","test_json_nstring":"12345"}
{"id":5,"test_boolean":null,"test_long":null,"test_double":null,"test_string":null,"test_timestamp":null,"test_json":"123.45","test_json_text":"123.45","test_json_string":"123.45","test_json_nstring":"123.45"}
{"id":6,"test_boolean":null,"test_long":null,"test_double":null,"test_string":null,"test_timestamp":null,"test_json":"\"hoge\"","test_json_text":"\"hoge\"","test_json_string":"\"hoge\"","test_json_nstring":"\"hoge\""}
2024-05-22 00:47:03.515 +0000 [INFO] (0001:transaction): {done:  1 / 1, running: 0}
2024-05-22 00:47:03.518 +0000 [INFO] (main): Committed.
2024-05-22 00:47:03.518 +0000 [INFO] (main): Next config diff: {"in":{},"out":{}}
config.yml (incremental) 
$ cat config.yml
in:
  type: sqlserver
  product: azure_synapse_analytics
  host: test-synapse-workspace0.sql.azuresynapse.net
  user: test_user@test-synapse-workspace0
  password: ***
  database: test_dedicated_sql_pool
  schema: test_schema
  table: test_table
  incremental: true
  incremental_columns: [id]
  column_options:
    test_timestamp: {type: string, timestamp_format: '%Y-%m-%d %H:%M:%S.%6N %z'}
out:
  type: command
  command: 'cat -'
  formatter:
    type: jsonl
run (incremental, first) 
$ embulk run config.yml -c diff.yml
2024-05-22 00:44:45.995 +0000: Embulk v0.9.26
2024-05-22 00:44:46.869 +0000 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
2024-05-22 00:44:48.520 +0000 [INFO] (main): Gem's home and path are set by default: "/home/ubuntu/.embulk/lib/gems"
2024-05-22 00:44:49.020 +0000 [INFO] (main): Started Embulk v0.9.26
2024-05-22 00:44:49.156 +0000 [INFO] (0001:transaction): Loaded plugin embulk-input-sqlserver (0.13.2)
2024-05-22 00:44:49.195 +0000 [INFO] (0001:transaction): Loaded plugin embulk-output-command (0.1.4)
2024-05-22 00:44:49.238 +0000 [WARN] (0001:transaction): "UTC" is recognized as "Z" to be compatible with the legacy style.
2024-05-22 00:44:49.259 +0000 [INFO] (0001:transaction): Connecting to jdbc:sqlserver://test-synapse-workspace0.sql.azuresynapse.net:1433 options {hostNameInCertificate=*.sql.azuresynapse.net, socketTimeout=1800000, user=test_user@test-synapse-workspace0, encrypt=true, databaseName=test_dedicated_sql_pool, trustServerCertificate=false, applicationName=embulk-input-sqlserver, password=***, loginTimeout=300}
2024-05-22 00:44:59.294 +0000 [INFO] (0001:transaction): Using JDBC Driver 7.2.2.0
2024-05-22 00:45:00.874 +0000 [INFO] (0001:transaction): Using local thread executor with max_threads=16 / output tasks 8 = input tasks 1 * 8
2024-05-22 00:45:00.915 +0000 [INFO] (0001:transaction): Loaded plugin embulk-formatter-jsonl (0.1.4)
2024-05-22 00:45:00.947 +0000 [INFO] (0001:transaction): {done:  0 / 1, running: 0}
2024-05-22 00:45:00.956 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:45:00.966 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:45:00.969 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:45:00.972 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:45:00.974 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:45:00.977 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:45:00.981 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:45:00.984 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:45:00.995 +0000 [WARN] (0015:task-0000): Z is deprecated as a military time zone name. Use UTC instead.
2024-05-22 00:45:00.995 +0000 [WARN] (0015:task-0000): "Z" is recognized as "Z" to be compatible with the legacy style.
2024-05-22 00:45:01.017 +0000 [INFO] (0015:task-0000): Connecting to jdbc:sqlserver://test-synapse-workspace0.sql.azuresynapse.net:1433 options {hostNameInCertificate=*.sql.azuresynapse.net, socketTimeout=1800000, user=test_user@test-synapse-workspace0, encrypt=true, databaseName=test_dedicated_sql_pool, trustServerCertificate=false, applicationName=embulk-input-sqlserver, password=***, loginTimeout=300}
2024-05-22 00:45:01.160 +0000 [INFO] (0015:task-0000): SQL: SELECT * FROM "test_schema"."test_table" ORDER BY "id"
2024-05-22 00:45:03.641 +0000 [INFO] (0015:task-0000): > 2.48 seconds
{"id":1,"test_boolean":true,"test_long":123,"test_double":1.23,"test_string":"あいうえお","test_timestamp":"1999-12-31 23:59:59.000000 +0000","test_json":"{\"キー\":\"値\"}","test_json_text":"{\"キー\":\"値\"}","test_json_string":"{\"キー\":\"値\"}","test_json_nstring":"{\"キー\":\"値\"}"}
{"id":2,"test_boolean":false,"test_long":456,"test_double":4.56,"test_string":"かきくけこ","test_timestamp":"2000-01-01 00:00:00.000000 +0000","test_json":"[{\"キー1\":\"値1\"},{\"キー2\":\"値2\"}]","test_json_text":"[{\"キー1\":\"値1\"},{\"キー2\":\"値2\"}]","test_json_string":"[{\"キー1\":\"値1\"},{\"キー2\":\"値2\"}]","test_json_nstring":"[{\"キー1\":\"値1\"},{\"キー2\":\"値2\"}]"}
{"id":3,"test_boolean":null,"test_long":null,"test_double":null,"test_string":null,"test_timestamp":null,"test_json":null,"test_json_text":null,"test_json_string":null,"test_json_nstring":null}
{"id":4,"test_boolean":null,"test_long":null,"test_double":null,"test_string":null,"test_timestamp":null,"test_json":"12345","test_json_text":"12345","test_json_string":"12345","test_json_nstring":"12345"}
{"id":5,"test_boolean":null,"test_long":null,"test_double":null,"test_string":null,"test_timestamp":null,"test_json":"123.45","test_json_text":"123.45","test_json_string":"123.45","test_json_nstring":"123.45"}
{"id":6,"test_boolean":null,"test_long":null,"test_double":null,"test_string":null,"test_timestamp":null,"test_json":"\"hoge\"","test_json_text":"\"hoge\"","test_json_string":"\"hoge\"","test_json_nstring":"\"hoge\""}
2024-05-22 00:45:03.753 +0000 [INFO] (0001:transaction): {done:  1 / 1, running: 0}
2024-05-22 00:45:03.758 +0000 [INFO] (main): Committed.
2024-05-22 00:45:03.759 +0000 [INFO] (main): Next config diff: {"in":{"last_record":[6]},"out":{}}
run (incremental, second) 
$ embulk run config.yml -c diff.yml
2024-05-22 00:45:10.268 +0000: Embulk v0.9.26
2024-05-22 00:45:10.920 +0000 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
2024-05-22 00:45:12.323 +0000 [INFO] (main): Gem's home and path are set by default: "/home/ubuntu/.embulk/lib/gems"
2024-05-22 00:45:12.847 +0000 [INFO] (main): Started Embulk v0.9.26
2024-05-22 00:45:12.911 +0000 [INFO] (0001:transaction): Loaded plugin embulk-input-sqlserver (0.13.2)
2024-05-22 00:45:12.938 +0000 [INFO] (0001:transaction): Loaded plugin embulk-output-command (0.1.4)
2024-05-22 00:45:12.967 +0000 [WARN] (0001:transaction): "UTC" is recognized as "Z" to be compatible with the legacy style.
2024-05-22 00:45:12.978 +0000 [INFO] (0001:transaction): Connecting to jdbc:sqlserver://test-synapse-workspace0.sql.azuresynapse.net:1433 options {hostNameInCertificate=*.sql.azuresynapse.net, socketTimeout=1800000, user=test_user@test-synapse-workspace0, encrypt=true, databaseName=test_dedicated_sql_pool, trustServerCertificate=false, applicationName=embulk-input-sqlserver, password=***, loginTimeout=300}
2024-05-22 00:45:13.351 +0000 [INFO] (0001:transaction): Using JDBC Driver 7.2.2.0
2024-05-22 00:45:13.491 +0000 [INFO] (0001:transaction): Using local thread executor with max_threads=16 / output tasks 8 = input tasks 1 * 8
2024-05-22 00:45:13.521 +0000 [INFO] (0001:transaction): Loaded plugin embulk-formatter-jsonl (0.1.4)
2024-05-22 00:45:13.546 +0000 [INFO] (0001:transaction): {done:  0 / 1, running: 0}
2024-05-22 00:45:13.554 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:45:13.565 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:45:13.568 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:45:13.570 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:45:13.573 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:45:13.577 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:45:13.581 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:45:13.586 +0000 [INFO] (0015:task-0000): Using command [sh, -c, cat -]
2024-05-22 00:45:13.592 +0000 [WARN] (0015:task-0000): Z is deprecated as a military time zone name. Use UTC instead.
2024-05-22 00:45:13.592 +0000 [WARN] (0015:task-0000): "Z" is recognized as "Z" to be compatible with the legacy style.
2024-05-22 00:45:13.620 +0000 [INFO] (0015:task-0000): Connecting to jdbc:sqlserver://test-synapse-workspace0.sql.azuresynapse.net:1433 options {hostNameInCertificate=*.sql.azuresynapse.net, socketTimeout=1800000, user=test_user@test-synapse-workspace0, encrypt=true, databaseName=test_dedicated_sql_pool, trustServerCertificate=false, applicationName=embulk-input-sqlserver, password=***, loginTimeout=300}
2024-05-22 00:45:13.765 +0000 [INFO] (0015:task-0000): SQL: SELECT * FROM "test_schema"."test_table" WHERE (("id" > ?)) ORDER BY "id"
2024-05-22 00:45:13.765 +0000 [INFO] (0015:task-0000): Parameters: [6]
2024-05-22 00:45:14.181 +0000 [INFO] (0015:task-0000): > 0.41 seconds
2024-05-22 00:45:14.201 +0000 [INFO] (0001:transaction): {done:  1 / 1, running: 0}
2024-05-22 00:45:14.205 +0000 [INFO] (main): Committed.
2024-05-22 00:45:14.206 +0000 [INFO] (main): Next config diff: {"in":{"last_record":[6]},"out":{}}

@t3t5u t3t5u requested a review from d-hrs May 20, 2024 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@d-hrs d-hrs Awaiting requested review from d-hrs

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@t3t5u