Proposal: Isolate Query Runners to Simplify Dependency Management

[arikfr]

I was thinking we could split out the less-used query runners — especially those with awkward or conflicting dependencies — into separate virtual environments inside the Docker container, and invoke them as subprocesses.

These runners could be bundled as independent packages or even standalone projects, with a very simple, well-defined protocol for execution. That would let us effectively stop maintaining their dependency trees in the main Redash codebase. Down the line, this also opens the door to non-Python runners if someone prefers that.

Upsides
• Much simpler dependency management for the core codebase
• Easier and safer automated dependency upgrades on a regular cadence
• Smaller memory footprint for the main processes

Downsides
• Added latency from spawning a subprocess per query
• Some upfront complexity in designing and wiring the protocol

Overall, this feels like a reasonable tradeoff to reduce long-term maintenance pain and isolate problematic dependencies.

[yoshiokatsuneo]

It looks a quite interesting idea !

At first, I think we can refer how other BI tools handle datasources.

• Superset
  It looks superset delegate dependency issue to SQLalchemy and DB-API python database driver.
  https://superset.apache.org/docs/configuration/databases/

• Metabase
  It looks Metabase provides JAR based plugin driver mechanism.
  https://www.metabase.com/docs/latest/developers-guide/community-drivers

And, I think that there are topics we can think about.

• Streaming v.s Store-and-Forward
  Currently, Redash stores all the result to the memory, and send it to client. If we just simply split query runner to two processes(main and sub), we may need double or triple memory space for storing the data(main process, sub process, and possibly communication pipe).

• Handling docker images for each runner.
  If we just create bunch of docker images for each data sources, the total size of the image may be huge. And, it make it a bit difficult to set up Redash environment as of now(and which is not very simple, yet.)

Anyway, we can start from rough sketch of the architecture, I think.

[yoshiokatsuneo]

@arikfr

note that it's not a separate container, but a separate virtual env.

Ah, I see. Thank you !

And, there are also some questions in my mind.

• Should we support the data sources which uses other virtual env ?
• Should we set up CI to run the test for the data sources with different virtual env ?

Thank you !

[arikfr]

Should we support the data sources which uses other virtual env ?

I think that we can let other people (who actually use them) support and maintain them. It's sort of like it is today, but more explicit.

Should we set up CI to run the test for the data sources with different virtual env ?

The nice thing is that we can test the integration point but don't have to test the individual packages.

[yoshiokatsuneo]

@arikfr

I think that we can let other people (who actually use them) support and maintain them. It's sort of like it is today, but more explicit.

In my understanding, as of today, all the data source is maintained by us, maintainers, and not other people. (As, at least one maintainer need to approve the PR.)

[arikfr]

Formally it's all maintained by us. But in practice, do you test every single one of them?

[yoshiokatsuneo]

@arikfr

For me, until now, I approved PRs only when I or other maintainer tested.
I feel it is not necessary and I'll approve that I have no testing environment. But at least I'll approve only when I fully understand the source code and feel good enough.

And, I'm just wondering who is the "other people".
Are you thinking to stop supporting data sources which have no "other people" who commit to maintain and develop the data source ?

For now, I think some data sources have intermittent contributor but no active maintainer. At now, I think those are somehow working, I'm just wondering we should stop supporting those data sources...