PromptTuner

An autonomous agent that writes, tests, and evolves its own prompts. This tool optimizes LLM prompts through iterative testing and refinement.

Features

• Autonomous Prompt Optimization: Generates prompts and iteratively improves them based on evaluation
• Configurable Evaluation: Uses a rubric to score prompt performance
• Fully Composable: Built on LangChain Runnable components
• Adaptable: Works for various task types including summarization, code generation, and data extraction

Architecture

 ┌───────────────────────────┐
 │  User Task (text + spec)  │
 └─────────────┬─────────────┘
               │
               ▼
 ┌───────────────────────────┐
 │  PromptGenerator (LLM)    │   «tool»
 └─────────────┬─────────────┘
               │ candidate prompt P
               ▼
 ┌───────────────────────────┐
 │  TaskRunner (LLM or chain)│   «tool»
 └─────────────┬─────────────┘
               │ answer A
               ▼
 ┌───────────────────────────┐
 │  Evaluator (LLM rubric)   │   «tool»
 └─────────────┬─────────────┘
               │ score S
               ▼
 ┌───────────────────────────┐
 │  PromptMutator            │   «pure TS fn»
 └─────────────┬─────────────┘
               │ new prompt P'
          loop until S ≥ threshold

Installation

# Clone the repository
git clone https://github.com/berkdurmus/promptuner.git
cd promptuner

# Install dependencies
npm install

# Build the project
npm run build

Usage

First, set your OpenAI API key:

# In .env file
OPENAI_API_KEY=your-api-key-here

# Or export in your shell
export OPENAI_API_KEY=your-api-key-here

Command Line

# Run the CLI
npm start -- tune -t "Summarize academic papers" -s "The summary should include key findings, methodology, and limitations."

# Options
npm start -- tune --help

Programmatic Usage

import { PromptTuner } from 'promptuner';

const task = {
  text: 'Generate SQL queries from natural language',
  spec: 'Queries should be valid PostgreSQL, handle complex joins, and include error handling comments',
};

const tuner = new PromptTuner({
  maxIterations: 5,
  scoreThreshold: 85,
});

const result = await tuner.tune(task);
console.log(`Best prompt (${result.finalScore}/100):`);
console.log(result.bestPrompt.text);

License

ISC