Skip to content

Add DB storage for events #31

@bhavya-tech

Description

@bhavya-tech

We have only FileEventSink for persistent event storage. However, it is only effective for local debugging of a small set of recorded events.
We need to add a proper DB support to store events so that frontend clients can access past events also.

Candidate databases for now:

  1. ClickHouse: It is a column store DB with best performance for append only writes and range queries.
    • There is chDB which is lightweight version. It can be used for local dev environment and small deployments.
    • It has native support of JSON also. This way Two columns can be timestamp and task name which will be used for efficient queries and other attributes can live in the JSON field.
    • Does not support direct modification effiicently. But does have batch delete/updates . Can be important for keeping particular time range data.
  2. TimescaleDB (PostgreSQL-based): Maturity of PostgreSQL, hybrid row and column hypertable.
    • Overall looks more mature than ClickHouse, but does not seem as specialised for our use case as ClickHouse.
  3. MongoDB: Native BSON support, widely used for JSON objects.
    • Good community support and integrations.
    • MongoDB Atlas gives free hosted servers for small projects.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions