Support Quarkus extension

[zhfeng]

Feature Request

I just want to add sharding sphere in Quarkus ecosystem and support as an extension

Is your feature request related to a problem?

Describe the feature you would like.

• create a quarkus extension just like quarkus-sharding-sphere
• It should leverage on Quarkus Datasource
• It needs to produce ShardingSphereDataSource beans with the application.properties or the yaml configuration.
• It nees to support in JVM and Native mode

Any feedback is welcome !

[tristaZero]

Hey @zhfeng ,

Your idea looks good to me. ;-) You want to add quarkus.datasource.db-kind=shardingsphere? Could I know why you want to add shardingsphere-db-kind for Quarkus?

I am curious which scenario people would like to choose Quarkus-dataSource?

[zhfeng]

yeah, that should be one option to add the db-kind of shardingshpere. I'd like to create a PoC poject in the next weeks.

It should leverage Quakrus native mode to build the shardingsphere applications into the native exeuctions by using Graal VM. This could improve the startup speed and memory in runtime.

Also I think if people want to build a cloud-native application with Quarkus framework and also want to use the database sharding features and shardingsphere should be their first and best choice. :)

[zhfeng]

I just create a initial repo https://github.com/zhfeng/quarkus-shardingsphere-jdbc. @tristaZero can you check the names are appropriate ? Currently Quarkus has https://github.com/quarkiverse/ to hold all of the 3rd party extensions. Anyway, we can also choose to hold the extension in shardingsphere repo.

[tristaZero]

Sounds interesting, I prefer to integrate into Quarkus ecosystem to make Quarkus users leverage ShardingSphere.
BTW, how do you think if ShardingSphere uses Quakus' dataSource in turn? I mean Vert.x of DataSource.
Look forward to your comment, your idea looks attractive. ;-)

[zhfeng]

It could depend on JdbcDataSourceBuildItem and create a Map<String, DataSource> which might be passed to ShardingSphereDataSourceFactory to create a new DataSource at runtime.

[zhfeng]

OK, I create https://github.com/quarkiverse/quarkus-shardingsphere-jdbc on Quarkus ecosystem.

[tristaZero]

Glad to see a new connection formed between two popular projects. @zhfeng

[zhfeng]

It seems that it's a little bit difficult to support the native mode right now. There are some dependencies required by shardingsphere nees to support graalvm compilation. I will keep investigate and gather some information here

• Upgarde to antltr 4.8 Upgrade antlr4 to 4.8 #12791
• apache calcite CALCITE-4786 Facilitate use of graalvm native-image compilation
  
• apache curator which might be some issues with create the client dynamically

[linghengqian]

• Handled by me at Add support for org.apache.curator:curator-*:5.4.0 oracle/graalvm-reachability-metadata#163 . But blocked by https://lists.apache.org/thread/stjp7nnhjp01129qfyqpwbxw2sylghzk .

[terrymanu]

apache curator is an optional dependency on ShardingSphere, how about use etcd to try it first?

[zhfeng]

OK, I get it. I'm trying to make a simple ShardingSphereDataSource to work at first. Btw, if curator is optional, why does the shardingsphere-jdbc-core depend on it ?

[linghengqian]

• Maybe I or other friends will try to commit Curator's GraalVM reachability metadata to https://github.com/oracle/graalvm-reachability-metadata . This issue is part of Make ShardingSphere Proxy in GraalVM Native Image form available #21347 .

[zhfeng]

@terrymanu @tristaZero I just wonder if we have some links to explain every shardingsphere configuration in application.properties with details ?

quarkus.shardingsphere.rules.sharding.default-sharding-column=account_id
quarkus.shardingsphere.rules.sharding.binding-tables=t_order,t_order_item
quarkus.shardingsphere.rules.sharding.broadcast-tables=t_address

I'd like to make sure these are the same with the shardingsphere -spring-boot-starter.

[tristaZero]

@zhfeng Sure, but I looks application.pros is supposed to be the same with shardingsphere -spring-boot-starter.

[tristaZero]

Do you think we need more work? Such as?

[zhfeng]

yeah, these properties should be the consistent with spring-boot-starter. I've added sharding-table properties and rules already and is working on

1. sharding database
2. read-write split

please review the ShardingTablesTest if you have a free time.

[linghengqian]

It looks like ShardingSphere doesn't actually work with GraalVM due to Apache Calcite ?
Aside from ShardingSphere Proxy, has anyone tried using Spring Native for SpringBoot projects with org.apache.shardingsphere:shardingsphere-jdbc-core-spring-boot-starter:5.0.0?

[linghengqian]

• Refer to https://twitter.com/rafaelcodes/status/1516494056532197379?t=p6lPYV4Fg73zu6sNU1TSkA&s=19 and Support subclass mocks on Graal VM. mockito/mockito#2613 , mockito 4.5.0 finally able to test on GraalVM native image, I think ShardingSphere unit tests on mockito can so start to try (of course, Improving mockito version from 3 to 4 is still a bit of work).

• update in 2022.08.01: Resolve in Update Mockito and ByteBuddy to help with GraalVM integration #17665 .

• update in 2022.08.01: Migrating some of the files processed by the heavy graalvm native build tools to https://github.com/oracle/graalvm-reachability-metadata may simplify some of the processing ideas.

[zhfeng]

@linghengqian are you interesting with quarkus-shardingsphere extension? please feel free to contribute on https://github.com/quarkiverse/quarkus-shardingsphere-jdbc

I do not have much time on it but I'd like to help you as much as possible. currently it failed to upgrade to 5.1.1

[linghengqian]

apache/calcite#2603 was closed due to controversy, I'm not sure about the impact, just a reminder.

[zhfeng]

Hi all,

I just finished work on quarkiverse/quarkus-shardingsphere-jdbc#56 with

• leverage on ShardingSphereDriver with 5.2.0
• add a db_kind with shardingsphere
• add a QuarkusDataSource to support get a datasource from the container.
• it works perfectly in JVM mode
• the native mode can not be supported right now since there are plenty building failures with graalvm native-image.

I still work on some document works and will relesae a first version of quarkus-shardingshpere-jdbc soon. If anybody from the community is intersted with quarkus and shardingsphere, feel free to have a try with this extension and any feedback is welcome!

Thanks a lot for the awesome work on shardingsphere 5.2.0 escpecially with supporting the SQL Driver, and this did a big help me for the working on the qurakus-shardingsphere-jdbc.

Cheers!
Zheng Feng

[zhfeng]

FYI, I done the first release of quarkus-shardingsphere-jdbc 0.0.1. If you are interested with it, please try and any feedback is welcome!
You need to do more configurations to make it work with quarkus-hibernate-orm. please check the docs!

[zhfeng]

Now, it can be found on https://code.quarkus.io/?extension-search=origin:other%20shardingsphere

[zhfeng]

I'm close to finish the native support in quarkus-shardingsphere-jdbc.

• Running groovy to parse the rules and store the results at build time.
• Add some subsitustions to make groovy working in GraalVM native.
• Register all SPI implementations in shardingsphere

Also we need some fixes in shardingsphere for helping native building

• Upgrade groovy to 4.0.10 #24732
• Refactor YamlLoggingRuleConfigurationSwapper to add getDefaultLoggingRuleConfiguration  #24728
• Refactor LoggingUtils to move out codes related to logback #24727
• Add jandex-maven-plugin to generate idx for classes #24726

I'd like to thank Nicolas Filotto @essobedo and I re-used his GroovySubstitutions.java on camel-quarkus

If the above fixes have been included in the coming release of 5.3.2, I'm glad to upgrade quarkus-shardingsphere-jdbc to support native building.

Cherers!