Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Database triggers #4860

Open
wants to merge 57 commits into
base: main
Choose a base branch
from
Open

Database triggers #4860

wants to merge 57 commits into from

Conversation

dieriba
Copy link
Collaborator

@dieriba dieriba commented Dec 5, 2024

PostgreSQL database trigger implementation that captures transaction events (INSERT, UPDATE, DELETE). Currently, the implementation identifies the type of transaction.

Currently working on decoding the trigger message to extract detailed information about the underlying data that caused the trigger to fire.


Important

Add support for managing database triggers in the frontend and backend, including new UI components and service methods.

  • Frontend:
    • Add DatabaseTriggersPanel to display database triggers in flows and scripts pages.
    • Add DatabaseTriggerEditor for creating and editing database triggers.
    • Update +layout.svelte to load database trigger usage.
    • Add database_triggers route with +page.svelte and +page.js for managing database triggers.
  • Backend:
    • Implement database trigger management in DatabaseTriggerService.
    • Add support for database triggers in getTriggersCountOfFlow and getTriggersCountOfScript.
  • Misc:
    • Update SidebarContent.svelte to include database triggers in the menu.
    • Add database trigger handling in TriggersBadge.svelte.

This description was created by Ellipsis for beed5fd. It will automatically update as commits are pushed.

Copy link
Contributor

github-actions bot commented Dec 5, 2024

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

Copy link

cloudflare-workers-and-pages bot commented Dec 7, 2024

Deploying windmill with  Cloudflare Pages  Cloudflare Pages

Latest commit: beed5fd
Status: ✅  Deploy successful!
Preview URL: https://6710650f.windmill.pages.dev
Branch Preview URL: https://database-triggers.windmill.pages.dev

View logs

@rubenfiszel rubenfiszel force-pushed the main branch 5 times, most recently from e36281c to f85cf46 Compare December 9, 2024 16:57
Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Looks good to me! Incremental review on 7011cda in 56 seconds

More details
  • Looked at 283 lines of code in 13 files
  • Skipped 2 files when reviewing.
  • Skipped posting 1 drafted comments based on config settings.
1. package.json:1
  • Draft comment:
    The package.json file is empty. Ensure it contains the necessary metadata and dependencies for the project.
  • Reason this comment was not posted:
    Comment did not seem useful.

Workflow ID: wflow_cZwmVncO4kWVN47y


You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

Copy link
Contributor

ellipsis-dev bot commented Dec 25, 2024

⚠️ This PR is too big for Ellipsis, but support for larger PRs is coming soon. If you want us to prioritize this feature, let us know at help@ellipsis.dev


Generated with ❤️ by ellipsis.dev

Copy link
Contributor

ellipsis-dev bot commented Dec 28, 2024

⚠️ This PR is too big for Ellipsis, but support for larger PRs is coming soon. If you want us to prioritize this feature, let us know at help@ellipsis.dev


Generated with ❤️ by ellipsis.dev

Copy link
Contributor

ellipsis-dev bot commented Dec 29, 2024

⚠️ This PR is too big for Ellipsis, but support for larger PRs is coming soon. If you want us to prioritize this feature, let us know at help@ellipsis.dev


Generated with ❤️ by ellipsis.dev

Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Looks good to me! Incremental review on eaf47f5 in 4 minutes and 7 seconds

More details
  • Looked at 9806 lines of code in 299 files
  • Skipped 0 files when reviewing.
  • Skipped posting 2 drafted comments based on config settings.
1. backend/.sqlx/query-0ef37117c369f03236e18f9dbb1f3d52776c8cb73f2507199c6ca16d4d2405ba.json:4
  • Draft comment:
    Using a leading wildcard '%' in a LIKE clause can cause performance issues if the column is not indexed properly. Consider reviewing the indexing strategy for the 'email' column.
  • Reason this comment was not posted:
    Confidence changes required: 50%
    The query in the JSON file uses a wildcard '%' in a LIKE clause, which can lead to performance issues if the column is not indexed properly. This is a common issue in SQL queries that use LIKE with leading wildcards.
2. backend/.sqlx/query-12a0fd7d8d99fb73b01bc24774fe9a8da57b5204bb6b1207aed47143c17a20bc.json:15
  • Draft comment:
    The nullable field for the COUNT result should be false instead of null as COUNT always returns a non-null value.
  • Reason this comment was not posted:
    Comment was on unchanged code.

Workflow ID: wflow_XuhP44EXSa8rdmDf


You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

Copy link
Contributor

ellipsis-dev bot commented Dec 30, 2024

⚠️ This PR is too big for Ellipsis, but support for larger PRs is coming soon. If you want us to prioritize this feature, let us know at help@ellipsis.dev


Generated with ❤️ by ellipsis.dev

Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Looks good to me! Incremental review on 66e305d in 1 minute and 59 seconds

More details
  • Looked at 12 lines of code in 1 files
  • Skipped 0 files when reviewing.
  • Skipped posting 1 drafted comments based on config settings.
1. backend/windmill-api/src/database_triggers/trigger.rs:42
  • Draft comment:
    The #[allow(unused)] attribute is unnecessary here as the RowExist trait is implemented and used. Consider removing it.
  • Reason this comment was not posted:
    Decided after close inspection that this draft comment was likely wrong and/or not actionable:
    While the trait is implemented, I don't see where this implementation is actually used in the code. The trait might be used in other files, but I can't see those. Without seeing the full usage context, I can't be certain that the #[allow(unused)] is unnecessary. The author may have added it for a reason.
    I could be wrong about the usage - there might be implicit uses of this trait that I'm not seeing, or it could be used through type inference. The implementation exists, which suggests it's used somewhere.
    Even with those possibilities, I don't have strong evidence that the attribute is unnecessary. The presence of an implementation doesn't guarantee the trait is actually used.
    Without being able to see the full context of how this trait is used across the codebase, I cannot confidently say that the #[allow(unused)] attribute is unnecessary.

Workflow ID: wflow_jEWh2OKighlJcOrf


You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

Copy link
Contributor

ellipsis-dev bot commented Jan 8, 2025

⚠️ This PR is too big for Ellipsis, but support for larger PRs is coming soon. If you want us to prioritize this feature, let us know at help@ellipsis.dev


Generated with ❤️ by ellipsis.dev

Copy link
Contributor

@HugoCasa HugoCasa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • we need to review the frontend/UX together, i'm not sure to understand everything
  • can you put the source for the code that was taken from pg_replicate? will make it easier to maintain

@@ -258,6 +261,7 @@ wasm-bindgen-test = "0.3.42"
convert_case = "0.6.0"
getrandom = "0.2"
tokio-postgres = {version = "^0.7", features = ["array-impls", "with-serde_json-1", "with-chrono-0_4", "with-uuid-1", "with-bit-vec-0_6"]}
rust-postgres = { package = "tokio-postgres", git = "https://github.com/imor/rust-postgres", rev = "20265ef38e32a06f76b6f9b678e2077fc2211f6b"}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you know why they use this specific version?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because it implement postgres copy both query which is needed to make the logical replication protocol works

backend/windmill-api/Cargo.toml Outdated Show resolved Hide resolved
backend/windmill-api/openapi.yaml Outdated Show resolved Hide resolved
frontend/src/lib/components/graph/util.ts Outdated Show resolved Hide resolved
backend/migrations/20241123152203_database_triggers.up.sql Outdated Show resolved Hide resolved
backend/migrations/20241123152203_database_triggers.up.sql Outdated Show resolved Hide resolved
backend/windmill-common/src/resource.rs Outdated Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better if put in another PR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're missing auth for a bunch of these APIs
also you should use authed to get the resource with get_resource_value_interpolated_internal

Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Looks good to me! Incremental review on c26c0f8 in 2 minutes and 19 seconds

More details
  • Looked at 1270 lines of code in 29 files
  • Skipped 1 files when reviewing.
  • Skipped posting 8 drafted comments based on config settings.
1. frontend/src/lib/components/GfmMarkdown.svelte:4
  • Draft comment:
    Consider adding a comment explaining that rehypeRaw and rehypeGithubAlerts are used as plugins for svelte-exmarkdown.
  • Reason this comment was not posted:
    Confidence changes required: 20%
    The code in frontend/src/lib/components/GfmMarkdown.svelte is importing rehypeRaw and rehypeGithubAlerts but does not use them directly. Instead, they are passed as plugins to svelte-exmarkdown. This is correct usage, but it might not be immediately clear to someone reading the code. A comment explaining this could be helpful.
2. frontend/src/lib/components/settings/ChangeWorkspaceColor.svelte:15
  • Draft comment:
    Consider adding a comment explaining the purpose of the reactive statement that calls generateRandomColor.
  • Reason this comment was not posted:
    Confidence changes required: 20%
    In frontend/src/lib/components/settings/ChangeWorkspaceColor.svelte, the function generateRandomColor is called in a reactive statement. This is correct, but it might not be immediately clear why this is done. A comment explaining the purpose of this reactive statement could be helpful.
3. frontend/src/lib/components/sidebar/MenuButton.svelte:14
  • Draft comment:
    Consider adding a comment explaining the purpose of the color prop and how it affects the button's styling.
  • Reason this comment was not posted:
    Confidence changes required: 20%
    In frontend/src/lib/components/sidebar/MenuButton.svelte, the color prop is used to conditionally apply styles. This is correct, but it might not be immediately clear why this is done. A comment explaining the purpose of the color prop could be helpful.
4. frontend/src/lib/components/sidebar/WorkspaceMenu.svelte:64
  • Draft comment:
    Consider adding a comment explaining the purpose of the color property and how it affects the workspace button's styling.
  • Reason this comment was not posted:
    Confidence changes required: 20%
    In frontend/src/lib/components/sidebar/WorkspaceMenu.svelte, the color property is used to style the workspace button. This is correct, but it might not be immediately clear why this is done. A comment explaining the purpose of the color property could be helpful.
5. frontend/src/lib/stores.ts:66
  • Draft comment:
    Consider adding a comment explaining the purpose of the color property in the userWorkspaces store.
  • Reason this comment was not posted:
    Confidence changes required: 20%
    In frontend/src/lib/stores.ts, the userWorkspaces store includes a color property. This is correct, but it might not be immediately clear why this property is included. A comment explaining the purpose of the color property could be helpful.
6. frontend/src/routes/(root)/(logged)/user/(user)/create_workspace/+page.svelte:43
  • Draft comment:
    Consider adding a comment explaining the purpose of the generateRandomColor function and when it is used.
  • Reason this comment was not posted:
    Confidence changes required: 20%
    In frontend/src/routes/(root)/(logged)/user/(user)/create_workspace/+page.svelte, the generateRandomColor function is used to set a random color for the workspace. This is correct, but it might not be immediately clear why this is done. A comment explaining the purpose of this function could be helpful.
7. frontend/src/routes/(root)/(logged)/user/(user)/workspaces/+page.svelte:31
  • Draft comment:
    Consider adding a comment explaining the purpose of the color property and how it affects the workspace button's styling.
  • Reason this comment was not posted:
    Confidence changes required: 20%
    In frontend/src/routes/(root)/(logged)/user/(user)/workspaces/+page.svelte, the color property is used to style the workspace button. This is correct, but it might not be immediately clear why this is done. A comment explaining the purpose of the color property could be helpful.
8. frontend/src/routes/(root)/(logged)/workspace_settings/+page.svelte:59
  • Draft comment:
    Consider adding a comment explaining the purpose of the ChangeWorkspaceColor component and how it is used.
  • Reason this comment was not posted:
    Confidence changes required: 20%
    In frontend/src/routes/(root)/(logged)/workspace_settings/+page.svelte, the ChangeWorkspaceColor component is used to change the color of a workspace. This is correct, but it might not be immediately clear why this component is included. A comment explaining the purpose of this component could be helpful.

Workflow ID: wflow_eFQq8cU1t8Ad8GXG


You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Looks good to me! Incremental review on 81d3ec8 in 3 minutes and 52 seconds

More details
  • Looked at 490 lines of code in 11 files
  • Skipped 1 files when reviewing.
  • Skipped posting 2 drafted comments based on config settings.
1. backend/windmill-api/src/database_triggers/trigger.rs:223
  • Draft comment:
    serde_json::from_value is used incorrectly here. resource is a String, not a serde_json::Value. Use serde_json::from_str instead.
  • Reason this comment was not posted:
    Decided after close inspection that this draft comment was likely wrong and/or not actionable:
    The comment assumes resource is a String, but looking at the code flow: 1) It comes from a SQL query 2) It's wrapped in Option and flattened 3) The value column in databases often stores JSON data 4) The code is trying to deserialize it into a Database struct. This suggests resource is actually a serde_json::Value, not a String.
    I could be wrong about the database schema - maybe the value column actually stores strings. I'm making assumptions about the schema.
    Even with that uncertainty, the code pattern of using from_value after a SQL query strongly suggests this is correct usage for deserializing from a JSON database column. The comment appears to be making incorrect assumptions.
    The comment should be deleted. The code is likely correct in using from_value to deserialize JSON data from a database column.
2. backend/windmill-api/src/database_triggers/handler.rs:246
  • Draft comment:
    The error handling here is too generic. Instead of converting all errors to Error::NotFound, consider propagating the original error to provide more context.
  • Reason this comment was not posted:
    Decided after close inspection that this draft comment was likely wrong and/or not actionable:
    The comment has a point - the code is losing error context by mapping all errors to NotFound. However, looking at the code more carefully, this seems intentional as the error message specifically states "Database resource do not exist". The function's purpose appears to be to verify resource existence, so mapping to NotFound is reasonable. The error handling matches the function's intent.
    I could be wrong about the function's intent - maybe there are important error cases being hidden. The error message grammar is also incorrect ("do not exist" vs "does not exist").
    While the grammar could be improved, the error handling itself appears intentional and appropriate for a resource lookup function. The comment's suggestion would make the error handling more complex without clear benefit.
    The comment should be deleted as the current error handling appears to be an intentional design choice appropriate for this resource lookup function.

Workflow ID: wflow_SSYeRwdbDhS1VowD


You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

@@ -210,13 +209,32 @@ async fn listen_to_transactions(
db: DB,
mut killpill_rx: tokio::sync::broadcast::Receiver<()>,
) -> Result<(), Error> {
let resource = get_database_resource(
&db,
let resource = sqlx::query_scalar!(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should also use get_resource_value_interpolated here. You can use fetch_api_authed with the trigger editor/email fields

Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Looks good to me! Incremental review on beed5fd in 1 minute and 40 seconds

More details
  • Looked at 1127 lines of code in 25 files
  • Skipped 0 files when reviewing.
  • Skipped posting 1 drafted comments based on config settings.
1. backend/.sqlx/query-ef6795d93423f98eea82eb18e6332580dc7f7a9e5a67026f8c0b3077f371fc62.json:9
  • Draft comment:
    The type_info for hash is Int8, which is unusual for a hash value. Consider using a more appropriate data type like Text or Bytea.
  • Reason this comment was not posted:
    Decided after close inspection that this draft comment was likely wrong and/or not actionable:
    Since this is an auto-generated file reflecting the actual database schema, suggesting a type change here won't be useful - the change would need to be made in the actual database schema, not this file. Additionally, while Int8 might be unusual for a hash, we don't have enough context about why this design choice was made. The hash could be a numeric hash or could have specific performance requirements.
    The comment raises a valid point about typical hash storage patterns. Maybe there's a real performance or data integrity issue here.
    Even if the concern is valid, this auto-generated file is not the right place to raise it. Schema design discussions should happen when the database schema is being modified.
    Delete the comment because it's targeting an auto-generated file that reflects the actual database schema, making it not actionable in this context.

Workflow ID: wflow_lXIU1UZF4lqkd2un


You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants