Parallel Execution of DuckDB Queries using iFrame #88

itsTalwar · 2024-08-26T08:52:19Z

High level Overview

With Parallel vs Without

…eat/parallel-exec

* Dynamic list * Clear memory and db * Type fix * Clean up and add parsing for JSON * Fix E2E * Remove unwanted file

* Remove hardcoding * Remove unwanted stuff * Remove log

* add: tests * update: path * update: mock

* Shared Array Paralle ecec * Remove migrations.json * Add logs for debugging

* Shared Array Paralle ecec * Remove migrations.json * Add logs for debugging * fix: app id * update: index html --------- Co-authored-by: itsTalwar <nt@devrev.ai>

* handle: json shared array * update: method

* Adding more comments and events * remove as much as possible * Fix types

* some cleanup * remove: unwanted change

benchmarking/src/app/dbm-context/parallel-memory-dbm-context.tsx

meerkat-browser-runner/src/app/app.tsx

meerkat-browser-runner/src/assets/.gitkeep

meerkat-dbm/package.json

meerkat-dbm/src/dbm/__test__/dbm-parallel.spec.ts

meerkat-dbm/src/dbm/__test__/dbm.spec.ts

meerkat-dbm/src/dbm/dbm-parallel/dbm-parallel.ts

meerkat-dbm/src/dbm/dbm-parallel/iframe-manager.ts

meerkat-dbm/src/dbm/dbm.ts

meerkat-dbm/src/window-communication/runner-types.ts

meerkat-dbm/src/window-communication/window-communication.ts

meerkat-dbm/src/utils/duck-type-convertor.ts

meerkat-dbm/src/window-communication/window-communication.ts

meerkat-dbm/src/utils/array-convertor.ts

benchmarking/src/app/file-loader/file-loader.tsx

meerkat-browser-runner/src/app/app.tsx

server.js

meerkat-dbm/src/window-communication/window-communication.ts

meerkat-browser-runner/src/app/app.tsx

meerkat-dbm/package.json

itsTalwar and others added 29 commits August 14, 2024 21:26

WIP Parallel exec

f030e31

Communication Ref

beabf09

IG manager

2455b28

WIP

b8839b4

WIP: file register

5ec3051

WIP: parallel memory manager

da424a4

WIP

9cff6ca

Get messages working

81e0c31

Little more fix

5b1fa7e

It works idk how

a5932c0

fix: duplicate query execution

09c797e

Late night work

926d74b

Merge back

21de9ff

some: cleanup

266aaaa

Added comment

0fc1bbc

Merge branch 'feat/parallel-exec' of github.com:devrev/meerkat into f…

d0c3456

…eat/parallel-exec

update: cache logic

e42e80f

Feat/parallel exec nik (#83)

dcd0853

* Dynamic list * Clear memory and db * Type fix * Clean up and add parsing for JSON * Fix E2E * Remove unwanted file

Remove hardcoding of origin (#85)

e9d9a22

* Remove hardcoding * Remove unwanted stuff * Remove log

test(parallel-dbm): add tests for parallel dbm (#84)

4415ba8

* add: tests * update: path * update: mock

SharedArrayBuffer via Single domain (#86)

afd7e7d

* Shared Array Paralle ecec * Remove migrations.json * Add logs for debugging

fix(parallel-dbm): fix target and app id (#87)

b401c99

* Shared Array Paralle ecec * Remove migrations.json * Add logs for debugging * fix: app id * update: index html --------- Co-authored-by: itsTalwar <nt@devrev.ai>

feat(parallel-dbm): convert json data to shared array (#90)

17e80d4

* handle: json shared array * update: method

feat: Clean types and integrate events with runners (#91)

8ab7274

* Adding more comments and events * remove as much as possible * Fix types

Reduce the size of iframe and remove unwanted file (#92)

a168a1c

handle: file loader buffer type (#94)

b56b0ec

chore(dbm): cleanup parallel dbm (#95)

e231514

* some cleanup * remove: unwanted change

fix: types

356614b

fix: type

75c4bc6

vpbs2 previously approved these changes Aug 26, 2024

View reviewed changes