Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Database triggers #4860

Open
wants to merge 57 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 40 commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
6315ace
feat: init database triggers
dieriba Nov 13, 2024
3a41d0c
Merge branch 'main' into database_triggers
dieriba Nov 17, 2024
aa298c8
feat: :sparkles: wip: database_triggers
dieriba Nov 17, 2024
80cc8e0
feat: :sparkles: add database triggers front view
dieriba Nov 17, 2024
3e367e4
feat: :construction: database_triggers
dieriba Nov 17, 2024
c70ca15
feat: :construction: add definition in yaml, updated backend code and…
dieriba Nov 17, 2024
68c3a3c
feat: :construction: updated migration file, update openapi.yml, upda…
dieriba Nov 17, 2024
bc3e19b
fix: struct rust
dieriba Nov 24, 2024
9a1ad78
feat: :construction: update migrate, database trigger backend functio…
dieriba Nov 24, 2024
fa456db
feat: :construction: add resource picker front, update backend function
dieriba Nov 26, 2024
cdb272c
feat: :construction: edit inner database inner, update triggers
dieriba Nov 26, 2024
3cdf0a2
feat: :construction: database_triggers
dieriba Nov 27, 2024
fabe469
feat: :construction: update openapi yaml, prettied websocker trigger
dieriba Nov 30, 2024
64e9938
feat: :construction: database_triggers
dieriba Dec 1, 2024
cdfad9c
feat: :construction: add resource module, update variable file, worki…
dieriba Dec 2, 2024
378b3ed
feat: :bug: working sqlx query
dieriba Dec 2, 2024
7300370
feat: :construction: fix query with sqlx, added main loop
dieriba Dec 3, 2024
9751ffe
feat: :construction: add new column database_trigg
dieriba Dec 3, 2024
939e4a1
feat: :passport_control: run jobs works
dieriba Dec 3, 2024
716c18e
feat: :construction: handling slot name and replication
dieriba Dec 3, 2024
8b8a3e7
feat: restructring triggers, decoding trigger message on work
dieriba Dec 5, 2024
a864b03
feat: :construction: database_trigger
dieriba Dec 7, 2024
18b96cc
feat: :construction:
dieriba Dec 8, 2024
23d942c
feat: :construction: converter done, work on custom script
dieriba Dec 11, 2024
da4264e
feat: :construction: multiple trigger
dieriba Dec 12, 2024
51c807a
feat: :construction: adding new argument function
dieriba Dec 12, 2024
0928c8e
feat: :construction: database_trigger
dieriba Dec 12, 2024
f8814a1
feat: :construction: add generate template for front, update script p…
dieriba Dec 16, 2024
e41965a
feat: :construction: template script fix bug, work on restructing bac…
dieriba Dec 16, 2024
0215d5a
feat: :construction: update autogenerated script, add persistence sta…
dieriba Dec 17, 2024
d85e25f
feat: :construction: update structure client
dieriba Dec 19, 2024
5813063
feat: :construction: rewrited crud function
dieriba Dec 20, 2024
86877af
feat: :construction: added publication handler
dieriba Dec 21, 2024
69dcf6d
feat: :construction: new ui finished
dieriba Dec 21, 2024
c0f624a
feat: :construction: added slot function hanlder finish front ux
dieriba Dec 22, 2024
c1b5c6d
feat: :sparkles: ux improvement done, backend logic done
dieriba Dec 22, 2024
351a6b7
feat: :bug: fix where clause
dieriba Dec 22, 2024
7cb0b6c
Merge branch 'main' into database_triggers
dieriba Dec 24, 2024
348922d
feat:
dieriba Dec 24, 2024
7011cda
feat:
dieriba Dec 25, 2024
e431b28
chore: update .sqlx and remove empty package
dieriba Dec 25, 2024
8d46b20
Merge branch 'main' into database_triggers
dieriba Dec 26, 2024
c30d032
fix: update publication and remove unneccessary print
dieriba Dec 28, 2024
f85ec3f
feat: finish converter to json, remove save button, remove unused crate
dieriba Dec 29, 2024
01b6ff3
feat: update migration for database trigger, fixed query, fixed front…
dieriba Dec 30, 2024
230034b
chore: update .sqlx
dieriba Dec 30, 2024
a0ca315
Merge branch 'main' into database_triggers
dieriba Dec 30, 2024
eaf47f5
chore: update sqlx
dieriba Dec 30, 2024
da30536
chore: .sqlx
dieriba Dec 30, 2024
66e305d
chore: add unused
dieriba Dec 30, 2024
681f193
Merge branch 'main' into database_triggers
dieriba Jan 7, 2025
4c05214
fix: use right database resource in back and front
dieriba Jan 8, 2025
d183b59
Merge branch 'main' into database_triggers
dieriba Jan 8, 2025
b44263c
fix build
HugoCasa Jan 9, 2025
c26c0f8
Merge remote-tracking branch 'origin/main' into database_triggers
HugoCasa Jan 9, 2025
81d3ec8
refactor:
dieriba Jan 10, 2025
beed5fd
fix sqlx
HugoCasa Jan 10, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion backend/.vscode/settings.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
{
"python.analysis.typeCheckingMode": "basic",
"rust-analyzer.linkedProjects": ["./windmill-common/Cargo.toml"],
"rust-analyzer.showUnlinkedFileNotification": false
"rust-analyzer.showUnlinkedFileNotification": false,
"conventionalCommits.scopes": [
"restructring triggers, decoding trigger message on work"
]
}
79 changes: 73 additions & 6 deletions backend/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 6 additions & 1 deletion backend/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,8 @@ serde.workspace = true
deno_core = { workspace = true, optional = true }
object_store = { workspace = true, optional = true }
quote.workspace = true
memchr.workspace = true


[target.'cfg(not(target_env = "msvc"))'.dependencies]
tikv-jemallocator = { optional = true, workspace = true }
Expand Down Expand Up @@ -152,6 +154,7 @@ windmill-parser-graphql = { path = "./parsers/windmill-parser-graphql" }
windmill-parser-php = { path = "./parsers/windmill-parser-php" }
windmill-api-client = { path = "./windmill-api-client" }

memchr = "2.7.4"
axum = { version = "^0.7", features = ["multipart"] }
headers = "^0"
hyper = { version = "^1", features = ["full"] }
Expand Down Expand Up @@ -258,6 +261,7 @@ wasm-bindgen-test = "0.3.42"
convert_case = "0.6.0"
getrandom = "0.2"
tokio-postgres = {version = "^0.7", features = ["array-impls", "with-serde_json-1", "with-chrono-0_4", "with-uuid-1", "with-bit-vec-0_6"]}
rust-postgres = { package = "tokio-postgres", git = "https://github.com/imor/rust-postgres", rev = "20265ef38e32a06f76b6f9b678e2077fc2211f6b"}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you know why they use this specific version?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because it implement postgres copy both query which is needed to make the logical replication protocol works

bit-vec = "=0.6.3"
mappable-rc = "^0"
mysql_async = { version = "*", default-features = false, features = ["minimal", "default", "native-tls-tls"]}
Expand Down Expand Up @@ -285,7 +289,7 @@ openssl = "=0.10"
mail-parser = "^0"
matchit = "=0.7.3"
rdkafka = { version = "0.36.2", features = ["cmake-build", "ssl-vendored"] }

pg_escape = "0.1.1"
datafusion = "39.0.0"
object_store = { version = "0.10.0", features = ["aws", "azure"] }
openidconnect = { version = "4.0.0-rc.1" }
Expand All @@ -307,6 +311,7 @@ opentelemetry-semantic-conventions = { version = "*", features = ["semconv_exper
bollard = "0.18.1"

tonic = { version = "^0", features = ["tls-native-roots"] }
byteorder = "1.5.0"

tikv-jemallocator = { version = "0.5" }
tikv-jemalloc-sys = { version = "^0.5" }
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
-- Add down migration script here
DROP TABLE IF EXISTS database_trigger;
DROP TYPE IF EXISTS transaction;
21 changes: 21 additions & 0 deletions backend/migrations/20241123152203_database_triggers.up.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
-- Add up migration script here
CREATE TYPE transaction AS ENUM ('Insert', 'Update', 'Delete');
dieriba marked this conversation as resolved.
Show resolved Hide resolved
CREATE TABLE database_trigger(
path VARCHAR(255) NOT NULL,
script_path VARCHAR(255) NOT NULL,
is_flow BOOLEAN NOT NULL,
workspace_id VARCHAR(50) NOT NULL,
edited_by VARCHAR(50) NOT NULL,
email VARCHAR(255) NOT NULL,
edited_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
extra_perms JSONB NULL,
database_resource_path VARCHAR(255) NOT NULL,
error TEXT NULL,
server_id VARCHAR(50) NULL,
last_server_ping TIMESTAMPTZ NULL,
replication_slot_name VARCHAR(255) NOT NULL,
publication_name VARCHAR(255) NOT NULL,
last_lsn bigint NOT NULL DEFAULT 0,
enabled BOOLEAN NOT NULL,
CONSTRAINT PK_database_trigger PRIMARY KEY (path,workspace_id)
dieriba marked this conversation as resolved.
Show resolved Hide resolved
);
Empty file removed backend/openapi-deref.yaml
Empty file.
6 changes: 5 additions & 1 deletion backend/windmill-api/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,12 @@ matchit = { workspace = true, optional = true }
tokio-tungstenite = { workspace = true, optional = true}
rdkafka = { workspace = true, optional = true }
const_format.workspace = true

rust-postgres.workspace = true
pin-project.workspace = true
http.workspace = true
async-stream.workspace = true
ulid.workspace = true
pg_escape.workspace = true
byteorder.workspace = true
memchr.workspace = true
thiserror.workspace = true
Loading