A Next.js & Effect powered AI pipeline that uploads, processes, and extracts metadata from images and PDFs using OpenAI, Tesseract.js, and AWS S3—fully async with Redis queue & Postgres storage.
flowchart TD;
User((User)) -->|Uploads Files| API[Next.js API: /api/upload]
API -->|Stores Files| S3[AWS S3]
API -->|Creates DB Record| DB[(PostgreSQL)]
API -->|Adds Job to Queue| Redis[Redis Queue]
subgraph Effect TS Worker
Worker[Job Processor] -->|Fetch Next Job| Redis
Worker -->|Download File| S3
Worker -->|Process File| AI[AI Processing]
AI -->|OpenAI Vision<br/>Tesseract.js OCR<br/>OpenAI Whisper| FileType
Worker -->|Save Results| DB
Worker -->|Mark Job Complete| DB
end
User -->|Checks Status| API2[Next.js API: /api/status]
API2 -->|Reads Status| DB
User -->|Fetches Results| API3[Next.js API: /api/result]
API3 -->|Returns Data| DB
| Tech | Purpose |
|---|---|
| Bun | Runtime |
| Next.js 15 | Frontend |
| Tailwind v4 | Styling |
| Effect TS | Async processing pipeline |
| AWS S3 | Object storage |
| Neon | Postgres DB |
| Redis | Caching & job queue |
| OpenAI Vision | Image analysis |
| Tesseract.js | PDF OCR |
- Install dependencies:
bun install- Set up environment variables:
cp .env.example .env- Run the development server:
bun devOpen http://localhost:3000 to view the application.
Required environment variables:
# Database
DATABASE_URL=
# Redis
REDIS_URL=
# AWS
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_REGION=
S3_BUCKET_NAME=
# OpenAI
OPENAI_API_KEY=POST /api/upload- Upload filesGET /api/status/:jobId- Check processing statusGET /api/result/:jobId- Retrieve processing results
# Run development server
bun dev
# Run tests
bun test
# Run type checking
bun type-check
# Run linting
bun lintThe application is designed to be deployed on Vercel with the following services:
| Provider | Service |
|---|---|
| Vercel | Frontend |
| Neon | Postgres |
| Upstash | Redis |
| AWS S3 | File Storage |
MIT