Skip to content

feat: add proto plugin for typed data contracts#220

Open
keugenek wants to merge 4 commits intomainfrom
feat/proto-plugin
Open

feat: add proto plugin for typed data contracts#220
keugenek wants to merge 4 commits intomainfrom
feat/proto-plugin

Conversation

@keugenek
Copy link
Copy Markdown
Contributor

@keugenek keugenek commented Mar 30, 2026

Summary

New AppKit plugin that provides protobuf-based serialization for typed data contracts across plugins, routes, and jobs. Consumers bring their own .proto files — AppKit provides the serialization layer.

Plugin API

Method Description
create(schema, init?) Create a typed proto message
serialize(schema, msg) Serialize to binary (Uint8Array)
deserialize(schema, data) Deserialize from binary
toJSON(schema, msg) Convert to JSON (proto field names)
fromJSON(schema, json) Parse from JSON

Usage

import { createApp, proto, files, server } from "@databricks/appkit";

const app = await createApp({
  plugins: [server(), proto(), files()],
});

const customer = app.proto.create(CustomerSchema, {
  id: "cust-001",
  name: "Acme Corp",
});

// Binary for storage — pair with Files plugin
const bytes = app.proto.serialize(CustomerSchema, customer);
await app.files("data").upload("customer.bin", Buffer.from(bytes));

// JSON for APIs
res.json(app.proto.toJSON(CustomerSchema, customer));

What's included

  • Proto plugin: packages/appkit/src/plugins/proto/ (plugin, serializer, types, manifest)
  • Unit tests: plugin + serializer
  • E2E scenario tests: Product Catalog sample app with Playwright
  • Documentation: docs/docs/plugins/proto.md

Design decisions

  • Pure serialization — no file I/O. Pair with Files plugin for Volume access, Lakebase plugin for database storage.
  • Zero configproto() with no arguments.
  • No bundled proto definitions — consumers bring their own .proto files and generate their own types. AppKit doesn't couple to any domain vocabulary.

Test plan

  • Plugin unit tests pass (5/5)
  • Serializer unit tests pass (6/6) — round-trip, nested objects, JSON, error handling
  • Exports only serialization API — no file I/O methods (writeToVolume, readFromVolume, exists are absent)
  • E2E scenario tests pass (13/13) — Product Catalog sample app
    • 5 public: view all, filter by category, health check, snake_case fields, API validation
    • 8 private: single-category, in/out-of-stock, product detail, 404, proto content-type, status text, column headers
    • Tested locally: npx tsx app/server.ts + npx playwright test
  • pnpm build passes
  • pnpm test passes (CI)

New plugin that provides protobuf-based serialization so plugins,
routes, and jobs share a single schema definition. Pairs with the
Files plugin for Volume I/O and Lakebase plugin for database storage.

Plugin API:
- create(schema, init?) — typed message construction
- serialize / deserialize — binary (Uint8Array)
- toJSON / fromJSON — JSON with proto field names

Also includes:
- Proto definitions: appkit/v1/common.proto (Error, Metadata, Value)
  and appkit/v1/pipeline.proto (JobResult, DataBatch, DataRow)
- Generated TypeScript types in packages/shared/src/proto/
- buf configuration for codegen
- Documentation with usage examples at docs/plugins/proto.md
- Unit tests for plugin and serializer

Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>
Co-authored-by: Isaac
Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>
Product Catalog sample app demonstrating proto plugin usage:
- Proto-style JSON serialization (snake_case field names)
- Binary endpoint with application/x-protobuf content type
- Category filtering with typed product contracts
- Playwright tests parameterized by public/private JSON cases

Runnable locally (npx tsx app/server.ts + playwright test) or
against a deployed app (APP_URL=... playwright test).

5 public cases (developer verification):
- View all products, filter by category, health check,
  API field validation, snake_case convention check

8 private cases (evaluation):
- Single-category filter, in/out-of-stock status, product detail,
  404 handling, proto content type, status text, column headers

Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>
Co-authored-by: Isaac
Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>
Fixes strict mode violation where "Stock" matched both "Stock" and
"In Stock" columns in Playwright tests. Also applies linter formatting.

Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>
The proto plugin provides serialization — consumers bring their own
.proto files and generate their own types. Shipping base types
(common.proto, pipeline.proto) in AppKit couples the SDK to specific
domain vocabulary that belongs in consumer projects.

Removed:
- proto/ top-level folder (buf config + .proto source files)
- packages/shared/src/proto/ (generated TypeScript types)
- turbo.json proto:generate task
- shared index proto re-export
- test-results/ artifact

Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant