Using Synapse with WhiteRabbit

[solmazeradat]

Hi,

Hope you are well.

I posted a similar post on the use of Databricks, Spark and Snowflake with whiteRabbit here.

We are looking at building a pipeline where the data volume/size is of the order of terabits. We want to ensure both the source data as well as the CDM data is compatible with analytical tools for big data as well as the OHDSI analytical tool kit.

Since the scanReport from WhiteRabbit tool is integral to the mapping process, wanted to check if WhiteRabbit supports the use of any of the following data databases in Synapse :

• PostgreSQL
• Data warehouse

Many thanks,
Solmaz

[janblom]

Hi,

I presume you are referring to the Azure Synapse Analytics platform, and I also assume that the Azure database connection option that is currently present in WhiteRabbit does not work with Synapse.

There is currently no support in WhiteRabbit for Synapse, and there are also no plans to implement this, as far as I know.

There are a few options though:

• you could add support for it yourself and make it available as a pull request. If you chose to do so, I strongly suggest to wait for the upcoming release with Snowflake support. Quite a bit of work has gone into factoring out the database specific code into a single class behind a generic interface, and this makes adding support for another database platform an easier and cleaner job. There is now also good support for automated testing of database connectivity.
• alternatively, if you are interested in sponsoring the work for a Synapse connector in WhiteRabbit, please contact me at The Hyve so that we can discuss the possibilities.

Best regards,
Jan Blom

[pasirikala]

This is not worth much but the Synapse Analytics dedicated instance database "kind of works" with the current WhiteRabbit:

• select PDW as the database (it is close enough to the Synapse Analytics)
• give the server location as "<server.databasedomain>;database=" (replacing <> with your connection details)
• ignore the warnings about USE statements 

Disclaimer: Some scan results work perfectly, others look ok but aren't accepted by Rabbit in a Hat (details are a bit foggy at this moment. Seems to be related to "<=" that is generated into the Fraction unique-column).